Hi Vineet, sounds indeed like a serialization issue - is the new field part of the object before it is serialized in the REST interface? In general, we use Jackson to serialize objects and there shouldn't be any additional configuration needed when using the JacksonSerializer from streampipes-serializers. Just as background information, we are slowly removing the RDF annotations from the framework and switch to plain JSON, so there shouldn't be any RDF anymore in the Core-UI communication. If you send me a code snippet or a link to the code, I'll be happy to have a look.
Dominik -----Original Message----- From: Vineet Tiwari <[email protected]> Sent: Monday, March 8, 2021 3:35 PM To: [email protected] Subject: Re: Intend to contribute to Streampipes Hey Guys, I picked up this issue https://issues.apache.org/jira/browse/STREAMPIPES-48 and started working on it. I was able to do most of the things. I am planning to reuse the EventPropertyNested instead of EventPropertyList for the lists containing objects. In order to differentiate nested objects and nested list, I have created another boolean property inside EventPropertyNested. I am able to see the nested properties displayed in the UI. The only issue is that I am not able to see the new property that I have added in the JSON response from the guess service. Because of this, I am not able to use it in the UI. I believe that there is some namespace library being used to serialise java to json objects. I even tried using the RDF annotations but I could not get it to work. I think there is some fundamental info that I am missing on this RDF library. Can someone please help me with this? Thanks, Vineet Tiwari On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <[email protected]> wrote: > Thanks Patrick. It is clear now. > Currently I am going through the backend code and understanding how > you are creating and running the pipeline graphs in the backend code. > Also I am going through some of the pipeline elements. > Do let me know if there are any issues I can look into. Anything that > you think may be suitable for me to look at. > > Thanks, > Vineet Tiwari > > On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <[email protected]> wrote: > >> Hi Vineet, >> >> nice to hear that you got it running. >> >> Regarding your questions: >> >> for 1) >> the options are up to you - „extensions-all-jvm“ is like a „fat“ >> bundle that contains everything. >> It basically is made of connect-adapter + pe-all-jvm. This is kinda >> experimental as we ship things in Docker images and as part of this >> option, we can reduce the overall system load in a single node setup >> due to less resource consumption for multiple JVM instances inside >> separate containers. >> >> However, for developing I’d suggest you start the adapters and >> pe-all-jvm separately. >> >> for 2) >> the connect-adapters service does the actual data connection to >> external data sources, e.g. kafka, mqtt, PLC as part of StreamPipes >> connect. Hence, this service runs the actually configured adapter >> instances in separate threads. Apart from that, in the >> setup/registration phase, the connect-adapter service tries to >> register at the backend and provides an adapter description which >> contains all the information such as what kind of protocols etc are >> supported by that service. Which is stored in Couchdb. >> At runtime, once you browse to the UI and got to the Connect adapter >> market place, the backend queries and endpoint of the connect-adapter >> service to check if is still actively running and besides retrieves >> the dedicated icons. >> >> Hope this helps. >> >> Patrick >> >> >> >> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <[email protected]>: >> >> Hey Patrick, >> Thanks for the clarifications. I was able to get things up and >> running in my machine. I have two more questions: >> >> 1. As you mentioned for the initialization of extensions, there are >> various projects to bundle them. What I gathered from the main class >> of these projects, I can see that extensions-all-jvm contains all >> elements (data set, stream, processor and sink). The others bundles >> contain subsets of these elements (For example: >> pipeline-elements-all-jvm contains all elements except data streams >> and data sets). To run the whole application, I need to run the >> extensions-all-jvm so that all the pipeline elements are initialized. >> Is my understanding correct? >> 2. When I navigate to the connect page of the application in UI, I >> see that backend tries to get icons for the elements in UI. To >> successfully get the icons, I need to run the connect-adapters. What >> exactly is the role of connect-adapters here apart from icons >> fetching. >> >> Thanks, >> Vineet Tiwari >> >> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected]> wrote: >> >> Hi Vineet, >> >> There are a few points here that will hopefully help you to get your >> dev setup running. >> >> Env variables >> We do recommend using IntelliJ as your IDE as we provide IntelliJ >> specific run configurations as part of the Github projects that will >> configure the services with appropriate env variables, e.g. for the >> core [1] and backend [2]. However, you can also run from Eclipse but >> you need to add these env variables manually to Eclipse. >> >> General Git project setup >> We currently differentiate between core and extensions git projects. >> The core contains core functionality for pipeline management, connect >> master, serialization, messaging wrapper, runtime wrapper etc. and >> two services that are also part of StreamPipes: >> >> - backend service (contains core functionality, e.g. pipeline >> management, connect master) >> - ui service >> >> It is worthy to note that we integrated the connect master service >> (was a standalone service before) in the backend service to simplify >> the overall architecture. This is why you did not find a main class >> for it anymore on the current dev branch. >> >> Besides, there exists the extensions project that contains all >> concrete connect adapters as well as pipeline elements. We can bundle >> them in various ways, e.g. >> >> - by type (streampipes-processors-filters-jvm) >> - by wrapper (streampipes-pipeline-elements-all-jvm) >> - all extensions by wrapper (streampipes-extensions-all-jvm) >> >> Dev environment >> We recommend checking out both projects and integrating them in a >> multi-project structure in your IDE. >> >> StreamPipes relies on a few external services, e.g. Consul, Couchdb, >> Kafka. To easify the development please use the StreamPipes CLI >> option to start your dev environment [3]. >> >> Here, there are various environments configurations depending on >> wether you want to develop: >> >> - core only -> backend >> - extensions only -> pipeline-element >> - core + extensions -> basic >> >> Go through the provided README file to see how to use the CLI. >> Briefly, it would be something like this for starting a docker dev >> environment with necessary services: >> >> ./streampipes env --set basic >> ./streampipes pull >> ./streampipes up -d >> >> In your IDE, set the the env variables for the services accordingly >> either using the provided run configs for IntelliJ or adding env >> variables for the StreamPipes to e.g. Eclipse by using the IntelliJ >> run configs as a template. >> >> We store this configs in Consul and do currently not overwrite them >> once a service has registered. >> You can either manually delete them from the Consul UI >> (localhost:8500) or simply start from a clean setup if you have a >> misconfiguration: >> >> ./streampipes clean -v >> >> NOTE! >> If non variables are present, all services use default variables that >> represent the prod settings when starting the containerized >> environment. Thus it is important to set these variables according to >> your setup. >> >> Networking in dev setup local machine <—> Docker This mixed setup: >> dev in IDE + Dockerized environment causes some network problems >> since use Consul (running in a container) to conduct service health >> checks to a service running on your local machine (in your IDE). >> Thus, we need an IP/DNS that is reachable/resolvable from inside the >> Docker container. >> >> Here it gets a little tricky as Docker currently does not provide a >> common solution for all OS. >> For instance, for the streampipes-pipeline-elements-all-jvm you will >> see in the run config that it sets an env variable as following [4]: >> >> <env name="SP_HOST" value="host.docker.internal" /> >> >> Here, SP_HOST refers to the IP/DNS of the service endpoint. >> Specifically, host.docker.internal is a Docker resolvable address on >> MacOS and Windows [5] >> >> For Linux: >> Whenever you see and env variable that uses „host.docker.internal“ as >> a value, you need to add the IP of you local machine >> >> —— >> >> Regarding your questions: >> >> 1. I was trying to start the backend module, which started fine, but >> when executing from UI, it is not able to connect to consul and mq >> due to host not found. I think I have not set the env variables >> correctly. I tried setting them in eclipse, but the host values are >> being taken from the consul key value store. How to change that to localhost? >> >> Related due to missing env variables when starting the backend main class. >> I assume, it’s trying to connect to something like consul:8500 >> instead of localhost:8500. >> Try with a clean setup, i.e., remove all docker volumes using the >> clean command above and setting the variables beforehand. >> >> >> 2. There are instructions to setup master container on the >> confluence, >> >> but >> >> I could not find its main class. >> >> This is most likely outdated, as explained above we recently >> integrated connect master into the backend. This is why you didn’t >> find a main class for the connect master. >> >> >> 3. Out of all the containers in docker, I am not sure of the function >> and code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to >> the backend api on http://backend:8030 by default. How do I change it >> to localhost? >> >> For the first part, I assume with function and code you mean where to >> find it? >> As said, we provide various bundles for pipeline elements. >> >> pipeline-element-all-jvm bundles all pipeline element modules of the >> extensions projects that are written for the plain Java wrapper (jvm) >> [6] (see the main class). >> >> connect-worker bundles all adapters [7] (see the main class). >> >> For the second part, this is mostly related again to missing env >> variables when starting the service. „backend“ is the default name of >> the docker service in prod. >> Take a look >> at the run configs for the env variables [8]. There you’ll find the >> following entry: >> >> <env name="SP_BACKEND_HOST" value="localhost" /> >> >> These env variables should be set in your IDE when starting your service. >> >> >> Hope this guide was helpful. >> >> We are currently restructuring the documentation, thus in the future >> we will add description as above right in the documentation. >> >> >> Patrick >> >> >> [1] >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon >> figurations >> < >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon >> figurations >> >> >> [2] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/. >> idea/runConfigurations >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/. >> idea/runConfigurations >> > >> >> [3] >> https://github.com/apache/incubator-streampipes-installer/tree/dev/cl >> i >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/c >> li> >> [4] >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon >> figurations/all_pipeline_elements_jvm.xml >> < >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon >> figurations/all_pipeline_elements_jvm.xml >> > >> >> [5] https://docs.docker.com/docker-for-windows/networking/ < >> https://docs.docker.com/docker-for-windows/networking/> >> [6] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-pipeline-elements-all-jvm >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-pipeline-elements-all-jvm >> > >> >> [7] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-connect-adapters >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-connect-adapters >> > >> >> [8] >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/. >> idea/runConfigurations/connect_adapters.xml >> < >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/. >> idea/runConfigurations/connect_adapters.xml >> > >> >> >> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: >> >> Hello, >> I have gone through some of the materials given above (thanks for >> that) >> >> and >> >> I have got a good idea on the high level architecture and design of >> the application. >> >> I started to set up streampipes on my local machine as well. >> I have the below queries: >> 1. I was trying to start the backend module, which started fine, but >> when executing from UI, it is not able to connect to consul and mq >> due to host not found. I think I have not set the env variables >> correctly. I tried setting them in eclipse, but the host values are >> being taken from the consul key value store. How to change that to localhost? >> 2. There are instructions to setup master container on the >> confluence, >> >> but >> >> I could not find its main class. >> 3. Out of all the containers in docker, I am not sure of the function >> and code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to >> the backend api on http://backend:8030 by default. How do I change it >> to localhost? >> >> Thanks, >> Vineet Tiwari >> >> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> >> >> wrote: >> >> >> Hi Vineet, >> >> welcome! >> If you have any questions on how to get started, we'll be glad to help. >> >> Doominik >> >> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: >> >> Hello, >> My name is Vineet Tiwari and I am working as a full time software >> developer. I am pretty good at Java and javascript (mostly with >> >> ReactJs) >> >> languages along with basic C++ and Python. >> I have a total of three and a half years of experience working as a >> SDE >> >> on >> >> several projects primarily with JAVA programming language. >> >> I do not have any prior open source contributions. I have just >> started >> >> to >> >> explore open-source projects and I found this project very >> >> interesting. I >> >> have gone through the website as well as the confluence page and I >> feel that I can surely contribute well to this project with my skills >> and capabilities. >> >> I have subscribed to the dev and commits mailing lists and I look >> >> forward >> >> to hearing from you about how I can contribute to the project. >> >> Thanks and regards >> Vineet Tiwari >> >> >>
