Hey Patrick, Thanks for the clarifications. I was able to get things up and running in my machine. I have two more questions:
1. As you mentioned for the initialization of extensions, there are various projects to bundle them. What I gathered from the main class of these projects, I can see that extensions-all-jvm contains all elements (data set, stream, processor and sink). The others bundles contain subsets of these elements (For example: pipeline-elements-all-jvm contains all elements except data streams and data sets). To run the whole application, I need to run the extensions-all-jvm so that all the pipeline elements are initialized. Is my understanding correct? 2. When I navigate to the connect page of the application in UI, I see that backend tries to get icons for the elements in UI. To successfully get the icons, I need to run the connect-adapters. What exactly is the role of connect-adapters here apart from icons fetching. Thanks, Vineet Tiwari On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected]> wrote: > Hi Vineet, > > There are a few points here that will hopefully help you to get your dev > setup running. > > Env variables > We do recommend using IntelliJ as your IDE as we > provide IntelliJ specific run configurations as part of the Github > projects that will configure > the services with appropriate env variables, e.g. for the core [1] and > backend [2]. However, > you can also run from Eclipse but you need to add these env variables > manually to Eclipse. > > General Git project setup > We currently differentiate between core and extensions git projects. The > core contains core > functionality for pipeline management, connect master, serialization, > messaging wrapper, > runtime wrapper etc. and two services that are also part of StreamPipes: > > - backend service (contains core functionality, e.g. pipeline management, > connect master) > - ui service > > It is worthy to note that we integrated the connect master service (was a > standalone service before) > in the backend service to simplify the overall architecture. This is why > you did not find a main > class for it anymore on the current dev branch. > > Besides, there exists the extensions project that contains all concrete > connect adapters as well > as pipeline elements. We can bundle them in various ways, e.g. > > - by type (streampipes-processors-filters-jvm) > - by wrapper (streampipes-pipeline-elements-all-jvm) > - all extensions by wrapper (streampipes-extensions-all-jvm) > > Dev environment > We recommend checking out both projects and integrating them in a > multi-project structure > in your IDE. > > StreamPipes relies on a few external services, e.g. Consul, Couchdb, > Kafka. To easify the > development please use the StreamPipes CLI option to start your dev > environment [3]. > > Here, there are various environments configurations depending on wether > you want to develop: > > - core only -> backend > - extensions only -> pipeline-element > - core + extensions -> basic > > Go through the provided README file to see how to use the CLI. Briefly, it > would be something > like this for starting a docker dev environment with necessary services: > > ./streampipes env --set basic > ./streampipes pull > ./streampipes up -d > > In your IDE, set the the env variables for the services accordingly either > using the provided > run configs for IntelliJ or adding env variables for the StreamPipes to > e.g. Eclipse by using > the IntelliJ run configs as a template. > > We store this configs in Consul and do currently not overwrite them once a > service has registered. > You can either manually delete them from the Consul UI (localhost:8500) or > simply start from > a clean setup if you have a misconfiguration: > > ./streampipes clean -v > > NOTE! > If non variables are present, all services use default variables that > represent the prod settings > when starting the containerized environment. Thus it is important to set > these variables > according to your setup. > > Networking in dev setup local machine <—> Docker > This mixed setup: dev in IDE + Dockerized environment causes some network > problems since > use Consul (running in a container) to conduct service health checks to a > service running on > your local machine (in your IDE). Thus, we need an IP/DNS that is > reachable/resolvable from > inside the Docker container. > > Here it gets a little tricky as Docker currently does not provide a common > solution for all OS. > For instance, for the streampipes-pipeline-elements-all-jvm you will see > in the run config that > it sets an env variable as following [4]: > > <env name="SP_HOST" value="host.docker.internal" /> > > Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically, > host.docker.internal > is a Docker resolvable address on MacOS and Windows [5] > > For Linux: > Whenever you see and env variable that uses „host.docker.internal“ as a > value, you need to > add the IP of you local machine > > —— > > Regarding your questions: > > > 1. I was trying to start the backend module, which started fine, but when > > executing from UI, it is not able to connect to consul and mq due to host > > not found. I think I have not set the env variables correctly. I tried > > setting them in eclipse, but the host values are being taken from the > > consul key value store. How to change that to localhost? > Related due to missing env variables when starting the backend main class. > I assume, it’s trying to connect to something like consul:8500 instead of > localhost:8500. > Try with a clean setup, i.e., remove all docker volumes using the clean > command > above and setting the variables beforehand. > > > > 2. There are instructions to setup master container on the confluence, > but > > I could not find its main class. > This is most likely outdated, as explained above we recently integrated > connect > master into the backend. This is why you didn’t find a main class for the > connect master. > > > > 3. Out of all the containers in docker, I am not sure of the function and > > code inside these two containers - pipeline-element-all and > connect-worker. > > Also, once started, the connect-worker keeps on trying to connect to the > > backend api on http://backend:8030 by default. How do I change it to > > localhost? > For the first part, I assume with function and code you mean where to find > it? > As said, we provide various bundles for pipeline elements. > > pipeline-element-all-jvm bundles all pipeline element modules of the > extensions > projects that are written for the plain Java wrapper (jvm) [6] (see the > main class). > > connect-worker bundles all adapters [7] (see the main class). > > For the second part, this is mostly related again to missing env variables > when starting > the service. „backend“ is the default name of the docker service in prod. > Take a look > at the run configs for the env variables [8]. There you’ll find the > following entry: > > <env name="SP_BACKEND_HOST" value="localhost" /> > > These env variables should be set in your IDE when starting your service. > > > Hope this guide was helpful. > > We are currently restructuring the documentation, thus in the future we > will add description as above > right in the documentation. > > > Patrick > > > [1] > https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations > < > https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations > > > [2] > https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations > < > https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations> > > [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli > <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli> > [4] > https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml > < > https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml> > > [5] https://docs.docker.com/docker-for-windows/networking/ < > https://docs.docker.com/docker-for-windows/networking/> > [6] > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm > < > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm> > > [7] > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters > < > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters> > > [8] > https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml > < > https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml> > > > > Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: > > > > Hello, > > I have gone through some of the materials given above (thanks for that) > and > > I have got a good idea on the high level architecture and design of the > > application. > > > > I started to set up streampipes on my local machine as well. > > I have the below queries: > > 1. I was trying to start the backend module, which started fine, but when > > executing from UI, it is not able to connect to consul and mq due to host > > not found. I think I have not set the env variables correctly. I tried > > setting them in eclipse, but the host values are being taken from the > > consul key value store. How to change that to localhost? > > 2. There are instructions to setup master container on the confluence, > but > > I could not find its main class. > > 3. Out of all the containers in docker, I am not sure of the function and > > code inside these two containers - pipeline-element-all and > connect-worker. > > Also, once started, the connect-worker keeps on trying to connect to the > > backend api on http://backend:8030 by default. How do I change it to > > localhost? > > > > Thanks, > > Vineet Tiwari > > > > On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> > wrote: > > > >> Hi Vineet, > >> > >> welcome! > >> If you have any questions on how to get started, we'll be glad to help. > >> > >> Doominik > >> > >> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: > >>> Hello, > >>> My name is Vineet Tiwari and I am working as a full time software > >>> developer. I am pretty good at Java and javascript (mostly with > ReactJs) > >>> languages along with basic C++ and Python. > >>> I have a total of three and a half years of experience working as a SDE > >> on > >>> several projects primarily with JAVA programming language. > >>> > >>> I do not have any prior open source contributions. I have just started > to > >>> explore open-source projects and I found this project very > interesting. I > >>> have gone through the website as well as the confluence page and I feel > >>> that I can surely contribute well to this project with my skills and > >>> capabilities. > >>> > >>> I have subscribed to the dev and commits mailing lists and I look > forward > >>> to hearing from you about how I can contribute to the project. > >>> > >>> Thanks and regards > >>> Vineet Tiwari > >>> > >> > >
