Hi Vineet, nice to hear that you got it running.
Regarding your questions: for 1) the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle that contains everything. It basically is made of connect-adapter + pe-all-jvm. This is kinda experimental as we ship things in Docker images and as part of this option, we can reduce the overall system load in a single node setup due to less resource consumption for multiple JVM instances inside separate containers. However, for developing I’d suggest you start the adapters and pe-all-jvm separately. for 2) the connect-adapters service does the actual data connection to external data sources, e.g. kafka, mqtt, PLC as part of StreamPipes connect. Hence, this service runs the actually configured adapter instances in separate threads. Apart from that, in the setup/registration phase, the connect-adapter service tries to register at the backend and provides an adapter description which contains all the information such as what kind of protocols etc are supported by that service. Which is stored in Couchdb. At runtime, once you browse to the UI and got to the Connect adapter market place, the backend queries and endpoint of the connect-adapter service to check if is still actively running and besides retrieves the dedicated icons. Hope this helps. Patrick > Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <[email protected]>: > > Hey Patrick, > Thanks for the clarifications. I was able to get things up and running in > my machine. I have two more questions: > > 1. As you mentioned for the initialization of extensions, there are various > projects to bundle them. What I gathered from the main class of these > projects, I can see that extensions-all-jvm contains all elements (data > set, stream, processor and sink). The others bundles contain subsets of > these elements (For example: pipeline-elements-all-jvm contains all > elements except data streams and data sets). To run the whole application, > I need to run the extensions-all-jvm so that all the pipeline elements are > initialized. Is my understanding correct? > 2. When I navigate to the connect page of the application in UI, I see that > backend tries to get icons for the elements in UI. To successfully get the > icons, I need to run the connect-adapters. What exactly is the role of > connect-adapters here apart from icons fetching. > > Thanks, > Vineet Tiwari > > On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected] > <mailto:[email protected]>> wrote: > >> Hi Vineet, >> >> There are a few points here that will hopefully help you to get your dev >> setup running. >> >> Env variables >> We do recommend using IntelliJ as your IDE as we >> provide IntelliJ specific run configurations as part of the Github >> projects that will configure >> the services with appropriate env variables, e.g. for the core [1] and >> backend [2]. However, >> you can also run from Eclipse but you need to add these env variables >> manually to Eclipse. >> >> General Git project setup >> We currently differentiate between core and extensions git projects. The >> core contains core >> functionality for pipeline management, connect master, serialization, >> messaging wrapper, >> runtime wrapper etc. and two services that are also part of StreamPipes: >> >> - backend service (contains core functionality, e.g. pipeline management, >> connect master) >> - ui service >> >> It is worthy to note that we integrated the connect master service (was a >> standalone service before) >> in the backend service to simplify the overall architecture. This is why >> you did not find a main >> class for it anymore on the current dev branch. >> >> Besides, there exists the extensions project that contains all concrete >> connect adapters as well >> as pipeline elements. We can bundle them in various ways, e.g. >> >> - by type (streampipes-processors-filters-jvm) >> - by wrapper (streampipes-pipeline-elements-all-jvm) >> - all extensions by wrapper (streampipes-extensions-all-jvm) >> >> Dev environment >> We recommend checking out both projects and integrating them in a >> multi-project structure >> in your IDE. >> >> StreamPipes relies on a few external services, e.g. Consul, Couchdb, >> Kafka. To easify the >> development please use the StreamPipes CLI option to start your dev >> environment [3]. >> >> Here, there are various environments configurations depending on wether >> you want to develop: >> >> - core only -> backend >> - extensions only -> pipeline-element >> - core + extensions -> basic >> >> Go through the provided README file to see how to use the CLI. Briefly, it >> would be something >> like this for starting a docker dev environment with necessary services: >> >> ./streampipes env --set basic >> ./streampipes pull >> ./streampipes up -d >> >> In your IDE, set the the env variables for the services accordingly either >> using the provided >> run configs for IntelliJ or adding env variables for the StreamPipes to >> e.g. Eclipse by using >> the IntelliJ run configs as a template. >> >> We store this configs in Consul and do currently not overwrite them once a >> service has registered. >> You can either manually delete them from the Consul UI (localhost:8500) or >> simply start from >> a clean setup if you have a misconfiguration: >> >> ./streampipes clean -v >> >> NOTE! >> If non variables are present, all services use default variables that >> represent the prod settings >> when starting the containerized environment. Thus it is important to set >> these variables >> according to your setup. >> >> Networking in dev setup local machine <—> Docker >> This mixed setup: dev in IDE + Dockerized environment causes some network >> problems since >> use Consul (running in a container) to conduct service health checks to a >> service running on >> your local machine (in your IDE). Thus, we need an IP/DNS that is >> reachable/resolvable from >> inside the Docker container. >> >> Here it gets a little tricky as Docker currently does not provide a common >> solution for all OS. >> For instance, for the streampipes-pipeline-elements-all-jvm you will see >> in the run config that >> it sets an env variable as following [4]: >> >> <env name="SP_HOST" value="host.docker.internal" /> >> >> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically, >> host.docker.internal >> is a Docker resolvable address on MacOS and Windows [5] >> >> For Linux: >> Whenever you see and env variable that uses „host.docker.internal“ as a >> value, you need to >> add the IP of you local machine >> >> —— >> >> Regarding your questions: >> >>> 1. I was trying to start the backend module, which started fine, but when >>> executing from UI, it is not able to connect to consul and mq due to host >>> not found. I think I have not set the env variables correctly. I tried >>> setting them in eclipse, but the host values are being taken from the >>> consul key value store. How to change that to localhost? >> Related due to missing env variables when starting the backend main class. >> I assume, it’s trying to connect to something like consul:8500 instead of >> localhost:8500. >> Try with a clean setup, i.e., remove all docker volumes using the clean >> command >> above and setting the variables beforehand. >> >> >>> 2. There are instructions to setup master container on the confluence, >> but >>> I could not find its main class. >> This is most likely outdated, as explained above we recently integrated >> connect >> master into the backend. This is why you didn’t find a main class for the >> connect master. >> >> >>> 3. Out of all the containers in docker, I am not sure of the function and >>> code inside these two containers - pipeline-element-all and >> connect-worker. >>> Also, once started, the connect-worker keeps on trying to connect to the >>> backend api on http://backend:8030 by default. How do I change it to >>> localhost? >> For the first part, I assume with function and code you mean where to find >> it? >> As said, we provide various bundles for pipeline elements. >> >> pipeline-element-all-jvm bundles all pipeline element modules of the >> extensions >> projects that are written for the plain Java wrapper (jvm) [6] (see the >> main class). >> >> connect-worker bundles all adapters [7] (see the main class). >> >> For the second part, this is mostly related again to missing env variables >> when starting >> the service. „backend“ is the default name of the docker service in prod. >> Take a look >> at the run configs for the env variables [8]. There you’ll find the >> following entry: >> >> <env name="SP_BACKEND_HOST" value="localhost" /> >> >> These env variables should be set in your IDE when starting your service. >> >> >> Hope this guide was helpful. >> >> We are currently restructuring the documentation, thus in the future we >> will add description as above >> right in the documentation. >> >> >> Patrick >> >> >> [1] >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations >> < >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations >> >> <https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations> >>> >> [2] >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations> >> < >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations>> >> >> [3] https://github.com/apache/incubator-streampipes-installer/tree/dev/cli >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli> >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli>> >> [4] >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml >> >> <https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml> >> < >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml >> >> <https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml>> >> >> [5] https://docs.docker.com/docker-for-windows/networking/ >> <https://docs.docker.com/docker-for-windows/networking/> < >> https://docs.docker.com/docker-for-windows/networking/ >> <https://docs.docker.com/docker-for-windows/networking/>> >> [6] >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm> >> < >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm>> >> >> [7] >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters> >> < >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters >> >> <https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters>> >> >> [8] >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml >> >> <https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml> >> < >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml >> >> <https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml>> >> >> >>> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: >>> >>> Hello, >>> I have gone through some of the materials given above (thanks for that) >> and >>> I have got a good idea on the high level architecture and design of the >>> application. >>> >>> I started to set up streampipes on my local machine as well. >>> I have the below queries: >>> 1. I was trying to start the backend module, which started fine, but when >>> executing from UI, it is not able to connect to consul and mq due to host >>> not found. I think I have not set the env variables correctly. I tried >>> setting them in eclipse, but the host values are being taken from the >>> consul key value store. How to change that to localhost? >>> 2. There are instructions to setup master container on the confluence, >> but >>> I could not find its main class. >>> 3. Out of all the containers in docker, I am not sure of the function and >>> code inside these two containers - pipeline-element-all and >> connect-worker. >>> Also, once started, the connect-worker keeps on trying to connect to the >>> backend api on http://backend:8030 by default. How do I change it to >>> localhost? >>> >>> Thanks, >>> Vineet Tiwari >>> >>> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> >> wrote: >>> >>>> Hi Vineet, >>>> >>>> welcome! >>>> If you have any questions on how to get started, we'll be glad to help. >>>> >>>> Doominik >>>> >>>> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: >>>>> Hello, >>>>> My name is Vineet Tiwari and I am working as a full time software >>>>> developer. I am pretty good at Java and javascript (mostly with >> ReactJs) >>>>> languages along with basic C++ and Python. >>>>> I have a total of three and a half years of experience working as a SDE >>>> on >>>>> several projects primarily with JAVA programming language. >>>>> >>>>> I do not have any prior open source contributions. I have just started >> to >>>>> explore open-source projects and I found this project very >> interesting. I >>>>> have gone through the website as well as the confluence page and I feel >>>>> that I can surely contribute well to this project with my skills and >>>>> capabilities. >>>>> >>>>> I have subscribed to the dev and commits mailing lists and I look >> forward >>>>> to hearing from you about how I can contribute to the project. >>>>> >>>>> Thanks and regards >>>>> Vineet Tiwari
smime.p7s
Description: S/MIME cryptographic signature
