Hi, in regarding to the SP_DATA_LOCATION configuration of the connect-adapters: It also can be removed and the path can be set automatically if the adapter is running in debug mode.
The only disadvantage is that the users cannot simply change the file locations of the uploaded files. Are there scenarios where this would be necessary? Johannes -----Ursprüngliche Nachricht----- Von: Dominik Riemer [mailto:[email protected]] Gesendet: Samstag, 16. November 2019 13:26 An: [email protected] Betreff: Re: AW: Run and develop StreamPipes without configuring the IP address Hi, cool, the config for pipeline elements looks good and is *much* easier to understand. For connect-adapters, can't we also remove SP_KAFKA_HOST (can be automatically set in debug mode similar to pipeline elements)? What about SP_DATA_LOCATION? For the backend config, we need to remove Kafka-Rest anyways (Confluent Community License) and AFAIK could also remove uri (RDF4J is now managed internally) and Zookeeper_HOST (not longer needed). Let's do some tests whether we can also just set a DEBUG flag when developing on the backend that automatically sets KAFKA_HOST, COUCHDB_HOST and JMS_HOST to localhost. Dominik On 2019/11/16 09:31:31, "Johannes Tex" <[email protected]> wrote: > Hi, > > > > I think we don't need any more these configurations for the backend : > > - SP_DATALAKE_HOST > > - SP_DATALAKE_PORT > > > > It was used for the first version of the data-lake with Elasticsearch , we > changed the data-lake database to Influx and removed the old data-lake API > with Elasticsearch last week. > > > > Furthermore I think it would be useful to add the port configuration for > Influx: > > SP_INFLUX_HOST=8086 > > > > > > Regards > > Johannes > > > > > > > > Von: Philipp Zehnder [mailto:[email protected]] > Gesendet: Samstag, 16. November 2019 09:57 > An: [email protected] > Betreff: Run and develop StreamPipes without configuring the IP address > > > > Hi all, > > > > we removed the requirement to define the servers IP address when StreamPipes > is run completely in Docker. > > However, we still have to change several configs to ensure the new setting > also works during development. > > > > This change effects the following projects: > > > > * streampipes > > * Do we still need all of those configurations for the backend? > > SP_COUCHDB_HOST=localhost > uri=http://localhost:8031/rdf4j-server > SP_KAFKA_HOST=host.docker.internal > SP_ZOOKEEPER_HOST=host.docker.internal > SP_JMS_HOST=host.docker.internal > SP_KAFKA_REST_HOST=host.docker.internal > SP_KAFKA_REST_PORT=8073 > SP_BACKEND_HOST=host.docker.internal > SP_ELASTICSEARCH_HOST=host.docker.internal > SP_DATALAKE_HOST=host.docker.internal > SP_DATALAKE_PORT=9200 > SP_INFLUX_HOST=host.docker.internal > > > > * streampipes-pipeline-elements > > * Suggestion of an example configuration for processing containers: > > > > SP_PORT=6060 > SP_HOST=host.docker.internal > SP_DEBUG=true > > > > * streampipes-connect-adapters > > * Suggestion of an example configuration for connect adapters > > > > SP_KAFKA_HOST=localhost > > SP_CONNECT_CONTAINER_HOST=localhost > > SP_CONNECT_CONTAINER_MASTER_HOST=localhost > > SP_CONNECT_CONTAINER_WORKER_HOST=localhost > > SP_DATA_LOCATION=./test_data/ > > SP_DEBUG=true > > > > > > Did I forget any configurations we have to change? > > > > I am still not completely certain when to use localhost and when to use > host.docker.internal during development. Currently I added > host.docker.internal to my /etc/hosts file. > > Is there a way to remove this entry? > > > > Additionally I suggest to remove all deployment folders (containing a > docker-compose.yml file) in the projects and manage them centrally in the > cli project. > > > > Cheers, > > Philipp > >
