Thanks Patrick. It is clear now. Currently I am going through the backend code and understanding how you are creating and running the pipeline graphs in the backend code. Also I am going through some of the pipeline elements. Do let me know if there are any issues I can look into. Anything that you think may be suitable for me to look at.
Thanks, Vineet Tiwari On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <[email protected]> wrote: > Hi Vineet, > > nice to hear that you got it running. > > Regarding your questions: > > for 1) > the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle > that contains everything. > It basically is made of connect-adapter + pe-all-jvm. This is kinda > experimental as we ship > things in Docker images and as part of this option, we can reduce the > overall system load in > a single node setup due to less resource consumption for multiple JVM > instances inside separate > containers. > > However, for developing I’d suggest you start the adapters and pe-all-jvm > separately. > > for 2) > the connect-adapters service does the actual data connection to external > data sources, e.g. kafka, mqtt, > PLC as part of StreamPipes connect. Hence, this service runs the actually > configured adapter > instances in separate threads. Apart from that, in the setup/registration > phase, the connect-adapter service > tries to register at the backend and provides an adapter description which > contains all the information > such as what kind of protocols etc are supported by that service. Which is > stored in Couchdb. > At runtime, once you browse to the UI and got to the Connect adapter > market place, the backend > queries and endpoint of the connect-adapter service to check if is still > actively running and besides retrieves > the dedicated icons. > > Hope this helps. > > Patrick > > > > Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <[email protected]>: > > Hey Patrick, > Thanks for the clarifications. I was able to get things up and running in > my machine. I have two more questions: > > 1. As you mentioned for the initialization of extensions, there are various > projects to bundle them. What I gathered from the main class of these > projects, I can see that extensions-all-jvm contains all elements (data > set, stream, processor and sink). The others bundles contain subsets of > these elements (For example: pipeline-elements-all-jvm contains all > elements except data streams and data sets). To run the whole application, > I need to run the extensions-all-jvm so that all the pipeline elements are > initialized. Is my understanding correct? > 2. When I navigate to the connect page of the application in UI, I see that > backend tries to get icons for the elements in UI. To successfully get the > icons, I need to run the connect-adapters. What exactly is the role of > connect-adapters here apart from icons fetching. > > Thanks, > Vineet Tiwari > > On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected]> wrote: > > Hi Vineet, > > There are a few points here that will hopefully help you to get your dev > setup running. > > Env variables > We do recommend using IntelliJ as your IDE as we > provide IntelliJ specific run configurations as part of the Github > projects that will configure > the services with appropriate env variables, e.g. for the core [1] and > backend [2]. However, > you can also run from Eclipse but you need to add these env variables > manually to Eclipse. > > General Git project setup > We currently differentiate between core and extensions git projects. The > core contains core > functionality for pipeline management, connect master, serialization, > messaging wrapper, > runtime wrapper etc. and two services that are also part of StreamPipes: > > - backend service (contains core functionality, e.g. pipeline management, > connect master) > - ui service > > It is worthy to note that we integrated the connect master service (was a > standalone service before) > in the backend service to simplify the overall architecture. This is why > you did not find a main > class for it anymore on the current dev branch. > > Besides, there exists the extensions project that contains all concrete > connect adapters as well > as pipeline elements. We can bundle them in various ways, e.g. > > - by type (streampipes-processors-filters-jvm) > - by wrapper (streampipes-pipeline-elements-all-jvm) > - all extensions by wrapper (streampipes-extensions-all-jvm) > > Dev environment > We recommend checking out both projects and integrating them in a > multi-project structure > in your IDE. > > StreamPipes relies on a few external services, e.g. Consul, Couchdb, > Kafka. To easify the > development please use the StreamPipes CLI option to start your dev > environment [3]. > > Here, there are various environments configurations depending on wether > you want to develop: > > - core only -> backend > - extensions only -> pipeline-element > - core + extensions -> basic > > Go through the provided README file to see how to use the CLI. Briefly, it > would be something > like this for starting a docker dev environment with necessary services: > > ./streampipes env --set basic > ./streampipes pull > ./streampipes up -d > > In your IDE, set the the env variables for the services accordingly either > using the provided > run configs for IntelliJ or adding env variables for the StreamPipes to > e.g. Eclipse by using > the IntelliJ run configs as a template. > > We store this configs in Consul and do currently not overwrite them once a > service has registered. > You can either manually delete them from the Consul UI (localhost:8500) or > simply start from > a clean setup if you have a misconfiguration: > > ./streampipes clean -v > > NOTE! > If non variables are present, all services use default variables that > represent the prod settings > when starting the containerized environment. Thus it is important to set > these variables > according to your setup. > > Networking in dev setup local machine <—> Docker > This mixed setup: dev in IDE + Dockerized environment causes some network > problems since > use Consul (running in a container) to conduct service health checks to a > service running on > your local machine (in your IDE). Thus, we need an IP/DNS that is > reachable/resolvable from > inside the Docker container. > > Here it gets a little tricky as Docker currently does not provide a common > solution for all OS. > For instance, for the streampipes-pipeline-elements-all-jvm you will see > in the run config that > it sets an env variable as following [4]: > > <env name="SP_HOST" value="host.docker.internal" /> > > Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically, > host.docker.internal > is a Docker resolvable address on MacOS and Windows [5] > > For Linux: > Whenever you see and env variable that uses „host.docker.internal“ as a > value, you need to > add the IP of you local machine > > —— > > Regarding your questions: > > 1. I was trying to start the backend module, which started fine, but when > executing from UI, it is not able to connect to consul and mq due to host > not found. I think I have not set the env variables correctly. I tried > setting them in eclipse, but the host values are being taken from the > consul key value store. How to change that to localhost? > > Related due to missing env variables when starting the backend main class. > I assume, it’s trying to connect to something like consul:8500 instead of > localhost:8500. > Try with a clean setup, i.e., remove all docker volumes using the clean > command > above and setting the variables beforehand. > > > 2. There are instructions to setup master container on the confluence, > > but > > I could not find its main class. > > This is most likely outdated, as explained above we recently integrated > connect > master into the backend. This is why you didn’t find a main class for the > connect master. > > > 3. Out of all the containers in docker, I am not sure of the function and > code inside these two containers - pipeline-element-all and > > connect-worker. > > Also, once started, the connect-worker keeps on trying to connect to the > backend api on http://backend:8030 by default. How do I change it to > localhost? > > For the first part, I assume with function and code you mean where to find > it? > As said, we provide various bundles for pipeline elements. > > pipeline-element-all-jvm bundles all pipeline element modules of the > extensions > projects that are written for the plain Java wrapper (jvm) [6] (see the > main class). > > connect-worker bundles all adapters [7] (see the main class). > > For the second part, this is mostly related again to missing env variables > when starting > the service. „backend“ is the default name of the docker service in prod. > Take a look > at the run configs for the env variables [8]. There you’ll find the > following entry: > > <env name="SP_BACKEND_HOST" value="localhost" /> > > These env variables should be set in your IDE when starting your service. > > > Hope this guide was helpful. > > We are currently restructuring the documentation, thus in the future we > will add description as above > right in the documentation. > > > Patrick > > > [1] > > https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations > < > > https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations > > > [2] > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations > < > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations > > > > [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli > <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli> > [4] > > https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml > < > > https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml > > > > [5] https://docs.docker.com/docker-for-windows/networking/ < > https://docs.docker.com/docker-for-windows/networking/> > [6] > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm > < > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm > > > > [7] > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters > < > > https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters > > > > [8] > > https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml > < > > https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml > > > > > Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: > > Hello, > I have gone through some of the materials given above (thanks for that) > > and > > I have got a good idea on the high level architecture and design of the > application. > > I started to set up streampipes on my local machine as well. > I have the below queries: > 1. I was trying to start the backend module, which started fine, but when > executing from UI, it is not able to connect to consul and mq due to host > not found. I think I have not set the env variables correctly. I tried > setting them in eclipse, but the host values are being taken from the > consul key value store. How to change that to localhost? > 2. There are instructions to setup master container on the confluence, > > but > > I could not find its main class. > 3. Out of all the containers in docker, I am not sure of the function and > code inside these two containers - pipeline-element-all and > > connect-worker. > > Also, once started, the connect-worker keeps on trying to connect to the > backend api on http://backend:8030 by default. How do I change it to > localhost? > > Thanks, > Vineet Tiwari > > On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> > > wrote: > > > Hi Vineet, > > welcome! > If you have any questions on how to get started, we'll be glad to help. > > Doominik > > On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: > > Hello, > My name is Vineet Tiwari and I am working as a full time software > developer. I am pretty good at Java and javascript (mostly with > > ReactJs) > > languages along with basic C++ and Python. > I have a total of three and a half years of experience working as a SDE > > on > > several projects primarily with JAVA programming language. > > I do not have any prior open source contributions. I have just started > > to > > explore open-source projects and I found this project very > > interesting. I > > have gone through the website as well as the confluence page and I feel > that I can surely contribute well to this project with my skills and > capabilities. > > I have subscribed to the dev and commits mailing lists and I look > > forward > > to hearing from you about how I can contribute to the project. > > Thanks and regards > Vineet Tiwari > > >
