Hi, thanks for your tips.
I tried to harmonize the configurations for StreamPipes during development.
Please try it out with your local IDE in case I missed something.
There are three different set up’s how you can run StreamPipes:
1. Run all in docker:
* This works now without changing anything
2. Develop PE’s / adapters in an IDE and run the rest in docker
* Users must set the environment variables according to the 'development/env'
file
* The rest should work as expected.
3. Develop backend in IDE
* To develop the backend the file ‘/etc/hosts’ must be edited
* Add the following line: '127.0.0.1 localhost host.docker.internal’
3.1 Start backend, ui, PE’s, connect-master, connect-workers in IDE
* Works without any further changes
3.2 Run backend + ui in IDE, PE’s and adapters in docker
* This setup currently does not work out of the box. Users have to set
several configurations like the couchdb, or hosts of PE’s / adapters manually
* For this setup it depends what the user starts in the IDE and what is
started in docker. Some services use the same configuration variables and they
must be configured accordingly.
3.3. Run backend in IDE, UI in docker
* Also requires additional configuration. The nginx server configurations
must be edited.
Locations of config files:
* backend (streampipes-backend/development/env)
* connect-Master (streampipes-connect-container/development/env)
* connect-worker-main (streampipes-connect-adapters/development/env)
* pipeline-elements: (in each module in the /development directory)
Additionally I deleted the deployment directories and renamed the .env files to
env (some IDE’s hide files with a leading dot).
Cheers,
Philipp
On 2019/11/16 09:31:31, "Johannes Tex" <[email protected]> wrote:
> Hi,
>
>
>
> I think we don't need any more these configurations for the backend :
>
> - SP_DATALAKE_HOST
>
> - SP_DATALAKE_PORT
>
>
>
> It was used for the first version of the data-lake with Elasticsearch , we
> changed the data-lake database to Influx and removed the old data-lake API
> with Elasticsearch last week.
>
>
>
> Furthermore I think it would be useful to add the port configuration for
> Influx:
>
> SP_INFLUX_HOST=8086
>
>
>
>
>
> Regards
>
> Johannes
>
>
>
>
>
>
>
> Von: Philipp Zehnder [mailto:[email protected]]
> Gesendet: Samstag, 16. November 2019 09:57
> An: [email protected]
> Betreff: Run and develop StreamPipes without configuring the IP address
>
>
>
> Hi all,
>
>
>
> we removed the requirement to define the servers IP address when StreamPipes
> is run completely in Docker.
>
> However, we still have to change several configs to ensure the new setting
> also works during development.
>
>
>
> This change effects the following projects:
>
>
>
> * streampipes
>
> * Do we still need all of those configurations for the backend?
>
> SP_COUCHDB_HOST=localhost
> uri=http://localhost:8031/rdf4j-server
> SP_KAFKA_HOST=host.docker.internal
> SP_ZOOKEEPER_HOST=host.docker.internal
> SP_JMS_HOST=host.docker.internal
> SP_KAFKA_REST_HOST=host.docker.internal
> SP_KAFKA_REST_PORT=8073
> SP_BACKEND_HOST=host.docker.internal
> SP_ELASTICSEARCH_HOST=host.docker.internal
> SP_DATALAKE_HOST=host.docker.internal
> SP_DATALAKE_PORT=9200
> SP_INFLUX_HOST=host.docker.internal
>
>
>
> * streampipes-pipeline-elements
>
> * Suggestion of an example configuration for processing containers:
>
>
>
> SP_PORT=6060
> SP_HOST=host.docker.internal
> SP_DEBUG=true
>
>
>
> * streampipes-connect-adapters
>
> * Suggestion of an example configuration for connect adapters
>
>
>
> SP_KAFKA_HOST=localhost
>
> SP_CONNECT_CONTAINER_HOST=localhost
>
> SP_CONNECT_CONTAINER_MASTER_HOST=localhost
>
> SP_CONNECT_CONTAINER_WORKER_HOST=localhost
>
> SP_DATA_LOCATION=./test_data/
>
> SP_DEBUG=true
>
>
>
>
>
> Did I forget any configurations we have to change?
>
>
>
> I am still not completely certain when to use localhost and when to use
> host.docker.internal during development. Currently I added
> host.docker.internal to my /etc/hosts file.
>
> Is there a way to remove this entry?
>
>
>
> Additionally I suggest to remove all deployment folders (containing a
> docker-compose.yml file) in the projects and manage them centrally in the
> cli project.
>
>
>
> Cheers,
>
> Philipp
>
>
.........................................................
M. Sc. Philipp Zehnder
Wissenschaftlicher Mitarbeiter | Research Scientist
Information Process Engineering (IPE)
FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14
76131 Karlsruhe, Germany
Tel.: +49 721 9654-805
Fax: +49 721 9654-806
[email protected] <mailto:[email protected]>
https://www.fzi.de/wir-ueber-uns/organisation/mitarbeiter/address/zehnder/
<https://www.fzi.de/wir-ueber-uns/organisation/mitarbeiter/address/zehnder/>
.........................................................
FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J.
Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
.........................................................
smime.p7s
Description: S/MIME cryptographic signature
