Hey Guys, I picked up this issue https://issues.apache.org/jira/browse/STREAMPIPES-48 and started working on it. I was able to do most of the things. I am planning to reuse the EventPropertyNested instead of EventPropertyList for the lists containing objects. In order to differentiate nested objects and nested list, I have created another boolean property inside EventPropertyNested. I am able to see the nested properties displayed in the UI. The only issue is that I am not able to see the new property that I have added in the JSON response from the guess service. Because of this, I am not able to use it in the UI. I believe that there is some namespace library being used to serialise java to json objects. I even tried using the RDF annotations but I could not get it to work. I think there is some fundamental info that I am missing on this RDF library. Can someone please help me with this?
Thanks, Vineet Tiwari On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <[email protected]> wrote: > Thanks Patrick. It is clear now. > Currently I am going through the backend code and understanding how you > are creating and running the pipeline graphs in the backend code. Also I am > going through some of the pipeline elements. > Do let me know if there are any issues I can look into. Anything that you > think may be suitable for me to look at. > > Thanks, > Vineet Tiwari > > On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <[email protected]> wrote: > >> Hi Vineet, >> >> nice to hear that you got it running. >> >> Regarding your questions: >> >> for 1) >> the options are up to you - „extensions-all-jvm“ is like a „fat“ bundle >> that contains everything. >> It basically is made of connect-adapter + pe-all-jvm. This is kinda >> experimental as we ship >> things in Docker images and as part of this option, we can reduce the >> overall system load in >> a single node setup due to less resource consumption for multiple JVM >> instances inside separate >> containers. >> >> However, for developing I’d suggest you start the adapters and pe-all-jvm >> separately. >> >> for 2) >> the connect-adapters service does the actual data connection to external >> data sources, e.g. kafka, mqtt, >> PLC as part of StreamPipes connect. Hence, this service runs the actually >> configured adapter >> instances in separate threads. Apart from that, in the setup/registration >> phase, the connect-adapter service >> tries to register at the backend and provides an adapter description >> which contains all the information >> such as what kind of protocols etc are supported by that service. Which >> is stored in Couchdb. >> At runtime, once you browse to the UI and got to the Connect adapter >> market place, the backend >> queries and endpoint of the connect-adapter service to check if is still >> actively running and besides retrieves >> the dedicated icons. >> >> Hope this helps. >> >> Patrick >> >> >> >> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <[email protected]>: >> >> Hey Patrick, >> Thanks for the clarifications. I was able to get things up and running in >> my machine. I have two more questions: >> >> 1. As you mentioned for the initialization of extensions, there are >> various >> projects to bundle them. What I gathered from the main class of these >> projects, I can see that extensions-all-jvm contains all elements (data >> set, stream, processor and sink). The others bundles contain subsets of >> these elements (For example: pipeline-elements-all-jvm contains all >> elements except data streams and data sets). To run the whole application, >> I need to run the extensions-all-jvm so that all the pipeline elements are >> initialized. Is my understanding correct? >> 2. When I navigate to the connect page of the application in UI, I see >> that >> backend tries to get icons for the elements in UI. To successfully get the >> icons, I need to run the connect-adapters. What exactly is the role of >> connect-adapters here apart from icons fetching. >> >> Thanks, >> Vineet Tiwari >> >> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected]> wrote: >> >> Hi Vineet, >> >> There are a few points here that will hopefully help you to get your dev >> setup running. >> >> Env variables >> We do recommend using IntelliJ as your IDE as we >> provide IntelliJ specific run configurations as part of the Github >> projects that will configure >> the services with appropriate env variables, e.g. for the core [1] and >> backend [2]. However, >> you can also run from Eclipse but you need to add these env variables >> manually to Eclipse. >> >> General Git project setup >> We currently differentiate between core and extensions git projects. The >> core contains core >> functionality for pipeline management, connect master, serialization, >> messaging wrapper, >> runtime wrapper etc. and two services that are also part of StreamPipes: >> >> - backend service (contains core functionality, e.g. pipeline management, >> connect master) >> - ui service >> >> It is worthy to note that we integrated the connect master service (was a >> standalone service before) >> in the backend service to simplify the overall architecture. This is why >> you did not find a main >> class for it anymore on the current dev branch. >> >> Besides, there exists the extensions project that contains all concrete >> connect adapters as well >> as pipeline elements. We can bundle them in various ways, e.g. >> >> - by type (streampipes-processors-filters-jvm) >> - by wrapper (streampipes-pipeline-elements-all-jvm) >> - all extensions by wrapper (streampipes-extensions-all-jvm) >> >> Dev environment >> We recommend checking out both projects and integrating them in a >> multi-project structure >> in your IDE. >> >> StreamPipes relies on a few external services, e.g. Consul, Couchdb, >> Kafka. To easify the >> development please use the StreamPipes CLI option to start your dev >> environment [3]. >> >> Here, there are various environments configurations depending on wether >> you want to develop: >> >> - core only -> backend >> - extensions only -> pipeline-element >> - core + extensions -> basic >> >> Go through the provided README file to see how to use the CLI. Briefly, it >> would be something >> like this for starting a docker dev environment with necessary services: >> >> ./streampipes env --set basic >> ./streampipes pull >> ./streampipes up -d >> >> In your IDE, set the the env variables for the services accordingly either >> using the provided >> run configs for IntelliJ or adding env variables for the StreamPipes to >> e.g. Eclipse by using >> the IntelliJ run configs as a template. >> >> We store this configs in Consul and do currently not overwrite them once a >> service has registered. >> You can either manually delete them from the Consul UI (localhost:8500) or >> simply start from >> a clean setup if you have a misconfiguration: >> >> ./streampipes clean -v >> >> NOTE! >> If non variables are present, all services use default variables that >> represent the prod settings >> when starting the containerized environment. Thus it is important to set >> these variables >> according to your setup. >> >> Networking in dev setup local machine <—> Docker >> This mixed setup: dev in IDE + Dockerized environment causes some network >> problems since >> use Consul (running in a container) to conduct service health checks to a >> service running on >> your local machine (in your IDE). Thus, we need an IP/DNS that is >> reachable/resolvable from >> inside the Docker container. >> >> Here it gets a little tricky as Docker currently does not provide a common >> solution for all OS. >> For instance, for the streampipes-pipeline-elements-all-jvm you will see >> in the run config that >> it sets an env variable as following [4]: >> >> <env name="SP_HOST" value="host.docker.internal" /> >> >> Here, SP_HOST refers to the IP/DNS of the service endpoint. Specifically, >> host.docker.internal >> is a Docker resolvable address on MacOS and Windows [5] >> >> For Linux: >> Whenever you see and env variable that uses „host.docker.internal“ as a >> value, you need to >> add the IP of you local machine >> >> —— >> >> Regarding your questions: >> >> 1. I was trying to start the backend module, which started fine, but when >> executing from UI, it is not able to connect to consul and mq due to host >> not found. I think I have not set the env variables correctly. I tried >> setting them in eclipse, but the host values are being taken from the >> consul key value store. How to change that to localhost? >> >> Related due to missing env variables when starting the backend main class. >> I assume, it’s trying to connect to something like consul:8500 instead of >> localhost:8500. >> Try with a clean setup, i.e., remove all docker volumes using the clean >> command >> above and setting the variables beforehand. >> >> >> 2. There are instructions to setup master container on the confluence, >> >> but >> >> I could not find its main class. >> >> This is most likely outdated, as explained above we recently integrated >> connect >> master into the backend. This is why you didn’t find a main class for the >> connect master. >> >> >> 3. Out of all the containers in docker, I am not sure of the function and >> code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to the >> backend api on http://backend:8030 by default. How do I change it to >> localhost? >> >> For the first part, I assume with function and code you mean where to find >> it? >> As said, we provide various bundles for pipeline elements. >> >> pipeline-element-all-jvm bundles all pipeline element modules of the >> extensions >> projects that are written for the plain Java wrapper (jvm) [6] (see the >> main class). >> >> connect-worker bundles all adapters [7] (see the main class). >> >> For the second part, this is mostly related again to missing env variables >> when starting >> the service. „backend“ is the default name of the docker service in prod. >> Take a look >> at the run configs for the env variables [8]. There you’ll find the >> following entry: >> >> <env name="SP_BACKEND_HOST" value="localhost" /> >> >> These env variables should be set in your IDE when starting your service. >> >> >> Hope this guide was helpful. >> >> We are currently restructuring the documentation, thus in the future we >> will add description as above >> right in the documentation. >> >> >> Patrick >> >> >> [1] >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations >> < >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runConfigurations >> >> >> [2] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/.idea/runConfigurations >> > >> >> [3] >> https://github.com/apache/incubator-streampipes-installer/tree/dev/cli >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/cli> >> [4] >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml >> < >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runConfigurations/all_pipeline_elements_jvm.xml >> > >> >> [5] https://docs.docker.com/docker-for-windows/networking/ < >> https://docs.docker.com/docker-for-windows/networking/> >> [6] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-pipeline-elements-all-jvm >> > >> >> [7] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/streampipes-connect-adapters >> > >> >> [8] >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml >> < >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/.idea/runConfigurations/connect_adapters.xml >> > >> >> >> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: >> >> Hello, >> I have gone through some of the materials given above (thanks for that) >> >> and >> >> I have got a good idea on the high level architecture and design of the >> application. >> >> I started to set up streampipes on my local machine as well. >> I have the below queries: >> 1. I was trying to start the backend module, which started fine, but when >> executing from UI, it is not able to connect to consul and mq due to host >> not found. I think I have not set the env variables correctly. I tried >> setting them in eclipse, but the host values are being taken from the >> consul key value store. How to change that to localhost? >> 2. There are instructions to setup master container on the confluence, >> >> but >> >> I could not find its main class. >> 3. Out of all the containers in docker, I am not sure of the function and >> code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to the >> backend api on http://backend:8030 by default. How do I change it to >> localhost? >> >> Thanks, >> Vineet Tiwari >> >> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> >> >> wrote: >> >> >> Hi Vineet, >> >> welcome! >> If you have any questions on how to get started, we'll be glad to help. >> >> Doominik >> >> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: >> >> Hello, >> My name is Vineet Tiwari and I am working as a full time software >> developer. I am pretty good at Java and javascript (mostly with >> >> ReactJs) >> >> languages along with basic C++ and Python. >> I have a total of three and a half years of experience working as a SDE >> >> on >> >> several projects primarily with JAVA programming language. >> >> I do not have any prior open source contributions. I have just started >> >> to >> >> explore open-source projects and I found this project very >> >> interesting. I >> >> have gone through the website as well as the confluence page and I feel >> that I can surely contribute well to this project with my skills and >> capabilities. >> >> I have subscribed to the dev and commits mailing lists and I look >> >> forward >> >> to hearing from you about how I can contribute to the project. >> >> Thanks and regards >> Vineet Tiwari >> >> >>
