Hi Vineet, There are a few points here that will hopefully help you to get your dev setup running.
Env variables We do recommend using IntelliJ as your IDE as we provide IntelliJ specific run configurations as part of the Github projects that will configure the services with appropriate env variables, e.g. for the core [1] and backend [2]. However, you can also run from Eclipse but you need to add these env variables manually to Eclipse. General Git project setup We currently differentiate between core and extensions git projects. The core contains core functionality for pipeline management, connect master, serialization, messaging wrapper, runtime wrapper etc. and two services that are also part of StreamPipes: - backend service (contains core functionality, e.g. pipeline management, connect master) - ui service It is worthy to note that we integrated the connect master service (was a standalone service before) in the backend service to simplify the overall architecture. This is why you did not find a main class for it anymore on the current dev branch. Besides, there exists the extensions project that contains all concrete connect adapters as well as pipeline elements. We can bundle them in various ways, e.g. - by type (streampipes-processors-filters-jvm) - by wrapper (streampipes-pipeline-elements-all-jvm) - all extensions by wrapper (streampipes-extensions-all-jvm) Dev environment We recommend checking out both projects and integrating them in a multi-project structure in your IDE. StreamPipes relies on a few external services, e.g. Consul, Couchdb, Kafka. To easify the development please use the StreamPipes CLI option to start your dev environment [3]. Here, there are various environments configurations depending on wether you want to develop: - core only -> backend - extensions only -> pipeline-element - core + extensions -> basic Go through the provided README file to see how to use the CLI. Briefly, it would be something like this for starting a docker dev environment with necessary services: ./streampipes env --set basic ./streampipes pull ./streampipes up -d In your IDE, set the the env variables for the services accordingly either using the provided run configs for IntelliJ or adding env variables for the StreamPipes to e.g. Eclipse by using the IntelliJ run configs as a template. We store this configs in Consul and do currently not overwrite them once a service has registered. You can either manually delete them from the Consul UI (localhost:8500) or simply start from a clean setup if you have a misconfiguration: ./streampipes clean -v NOTE! If non variables are present, all services use default variables that represent the prod settings when starting the containerized environment. Thus it is important to set these variables according to your setup. Networking in dev setup local machine <—> Docker This mixed setup: dev in IDE + Dockerized environment causes some network problems since use Consul (running in a container) to conduct service health checks to a service running on your local machine (in your IDE). Thus, we need an IP/DNS that is reachable/resolvable from inside the Docker container. Here it gets a little tricky as Docker currently does not provide a common solution for all OS. For instance, for the streampipes-pipeline-elements-all-jvm you will see in the run config that it sets an env variable as following [4]: <env name="SP_HOST" value="host.docker.internal" /> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically, host.docker.internal is a Docker resolvable address on MacOS and Windows [5] For Linux: Whenever you see and env variable that uses „host.docker.internal“ as a value, you need to add the IP of you local machine —— Regarding your questions: > 1. I was trying to start the backend module, which started fine, but when > executing from UI, it is not able to connect to consul and mq due to host > not found. I think I have not set the env variables correctly. I tried > setting them in eclipse, but the host values are being taken from the > consul key value store. How to change that to localhost? Related due to missing env variables when starting the backend main class. I assume, it’s trying to connect to something like consul:8500 instead of localhost:8500. Try with a clean setup, i.e., remove all docker volumes using the clean command above and setting the variables beforehand. > 2. There are instructions to setup master container on the confluence, but > I could not find its main class. This is most likely outdated, as explained above we recently integrated connect master into the backend. This is why you didn’t find a main class for the connect master. > 3. Out of all the containers in docker, I am not sure of the function and > code inside these two containers - pipeline-element-all and connect-worker. > Also, once started, the connect-worker keeps on trying to connect to the > backend api on http://backend:8030 by default. How do I change it to > localhost? For the first part, I assume with function and code you mean where to find it? As said, we provide various bundles for pipeline elements. pipeline-element-all-jvm bundles all pipeline element modules of the extensions projects that are written for the plain Java wrapper (jvm) [6] (see the main class). connect-worker bundles all adapters [7] (see the main class). For the second part, this is mostly related again to missing env variables when starting the service. „backend“ is the default name of the docker service in prod. Take a look at the run configs for the env variables [8]. There you’ll find the following entry: <env name="SP_BACKEND_HOST" value="localhost" /> These env variables should be set in your IDE when starting your service. Hope this guide was helpful. We are currently restructuring the documentation, thus in the future we will add description as above right in the documentation. Patrick [1] https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations <https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations> [2] https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations <https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations> [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli> [4] https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml <https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml> [5] https://docs.docker.com/docker-for-windows/networking/ <https://docs.docker.com/docker-for-windows/networking/> [6] https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm> [7] https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters> [8] https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml <https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml> > Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: > > Hello, > I have gone through some of the materials given above (thanks for that) and > I have got a good idea on the high level architecture and design of the > application. > > I started to set up streampipes on my local machine as well. > I have the below queries: > 1. I was trying to start the backend module, which started fine, but when > executing from UI, it is not able to connect to consul and mq due to host > not found. I think I have not set the env variables correctly. I tried > setting them in eclipse, but the host values are being taken from the > consul key value store. How to change that to localhost? > 2. There are instructions to setup master container on the confluence, but > I could not find its main class. > 3. Out of all the containers in docker, I am not sure of the function and > code inside these two containers - pipeline-element-all and connect-worker. > Also, once started, the connect-worker keeps on trying to connect to the > backend api on http://backend:8030 by default. How do I change it to > localhost? > > Thanks, > Vineet Tiwari > > On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> wrote: > >> Hi Vineet, >> >> welcome! >> If you have any questions on how to get started, we'll be glad to help. >> >> Doominik >> >> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: >>> Hello, >>> My name is Vineet Tiwari and I am working as a full time software >>> developer. I am pretty good at Java and javascript (mostly with ReactJs) >>> languages along with basic C++ and Python. >>> I have a total of three and a half years of experience working as a SDE >> on >>> several projects primarily with JAVA programming language. >>> >>> I do not have any prior open source contributions. I have just started to >>> explore open-source projects and I found this project very interesting. I >>> have gone through the website as well as the confluence page and I feel >>> that I can surely contribute well to this project with my skills and >>> capabilities. >>> >>> I have subscribed to the dev and commits mailing lists and I look forward >>> to hearing from you about how I can contribute to the project. >>> >>> Thanks and regards >>> Vineet Tiwari >>> >>
