Hi Vineet, just two additions that might help to better understand our serialization layer: - REST interfaces can be annotated with the @JacksonSerializer annotation which ensures that our custom serializer is used. - We auto-generate typescript objects from models that are annotated with the @TsModel annotation, there is some information on how to update the generated TS classes in the wiki at [1].
Hope this helps! Dominik [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Core-UI+Serialization -----Original Message----- From: Vineet Tiwari <[email protected]> Sent: Monday, March 8, 2021 3:35 PM To: [email protected] Subject: Re: Intend to contribute to Streampipes Hey Guys, I picked up this issue https://issues.apache.org/jira/browse/STREAMPIPES-48 and started working on it. I was able to do most of the things. I am planning to reuse the EventPropertyNested instead of EventPropertyList for the lists containing objects. In order to differentiate nested objects and nested list, I have created another boolean property inside EventPropertyNested. I am able to see the nested properties displayed in the UI. The only issue is that I am not able to see the new property that I have added in the JSON response from the guess service. Because of this, I am not able to use it in the UI. I believe that there is some namespace library being used to serialise java to json objects. I even tried using the RDF annotations but I could not get it to work. I think there is some fundamental info that I am missing on this RDF library. Can someone please help me with this? Thanks, Vineet Tiwari On Wed, Mar 3, 2021 at 2:07 PM Vineet Tiwari <[email protected]> wrote: > Thanks Patrick. It is clear now. > Currently I am going through the backend code and understanding how > you are creating and running the pipeline graphs in the backend code. > Also I am going through some of the pipeline elements. > Do let me know if there are any issues I can look into. Anything that > you think may be suitable for me to look at. > > Thanks, > Vineet Tiwari > > On Tue, Mar 2, 2021 at 7:20 PM Patrick Wiener <[email protected]> wrote: > >> Hi Vineet, >> >> nice to hear that you got it running. >> >> Regarding your questions: >> >> for 1) >> the options are up to you - „extensions-all-jvm“ is like a „fat“ >> bundle that contains everything. >> It basically is made of connect-adapter + pe-all-jvm. This is kinda >> experimental as we ship things in Docker images and as part of this >> option, we can reduce the overall system load in a single node setup >> due to less resource consumption for multiple JVM instances inside >> separate containers. >> >> However, for developing I’d suggest you start the adapters and >> pe-all-jvm separately. >> >> for 2) >> the connect-adapters service does the actual data connection to >> external data sources, e.g. kafka, mqtt, PLC as part of StreamPipes >> connect. Hence, this service runs the actually configured adapter >> instances in separate threads. Apart from that, in the >> setup/registration phase, the connect-adapter service tries to >> register at the backend and provides an adapter description which >> contains all the information such as what kind of protocols etc are >> supported by that service. Which is stored in Couchdb. >> At runtime, once you browse to the UI and got to the Connect adapter >> market place, the backend queries and endpoint of the connect-adapter >> service to check if is still actively running and besides retrieves >> the dedicated icons. >> >> Hope this helps. >> >> Patrick >> >> >> >> Am 02.03.2021 um 14:06 schrieb Vineet Tiwari <[email protected]>: >> >> Hey Patrick, >> Thanks for the clarifications. I was able to get things up and >> running in my machine. I have two more questions: >> >> 1. As you mentioned for the initialization of extensions, there are >> various projects to bundle them. What I gathered from the main class >> of these projects, I can see that extensions-all-jvm contains all >> elements (data set, stream, processor and sink). The others bundles >> contain subsets of these elements (For example: >> pipeline-elements-all-jvm contains all elements except data streams >> and data sets). To run the whole application, I need to run the >> extensions-all-jvm so that all the pipeline elements are initialized. >> Is my understanding correct? >> 2. When I navigate to the connect page of the application in UI, I >> see that backend tries to get icons for the elements in UI. To >> successfully get the icons, I need to run the connect-adapters. What >> exactly is the role of connect-adapters here apart from icons >> fetching. >> >> Thanks, >> Vineet Tiwari >> >> On Sun, Feb 28, 2021 at 5:57 PM Patrick Wiener <[email protected]> wrote: >> >> Hi Vineet, >> >> There are a few points here that will hopefully help you to get your >> dev setup running. >> >> Env variables >> We do recommend using IntelliJ as your IDE as we provide IntelliJ >> specific run configurations as part of the Github projects that will >> configure the services with appropriate env variables, e.g. for the >> core [1] and backend [2]. However, you can also run from Eclipse but >> you need to add these env variables manually to Eclipse. >> >> General Git project setup >> We currently differentiate between core and extensions git projects. >> The core contains core functionality for pipeline management, connect >> master, serialization, messaging wrapper, runtime wrapper etc. and >> two services that are also part of StreamPipes: >> >> - backend service (contains core functionality, e.g. pipeline >> management, connect master) >> - ui service >> >> It is worthy to note that we integrated the connect master service >> (was a standalone service before) in the backend service to simplify >> the overall architecture. This is why you did not find a main class >> for it anymore on the current dev branch. >> >> Besides, there exists the extensions project that contains all >> concrete connect adapters as well as pipeline elements. We can bundle >> them in various ways, e.g. >> >> - by type (streampipes-processors-filters-jvm) >> - by wrapper (streampipes-pipeline-elements-all-jvm) >> - all extensions by wrapper (streampipes-extensions-all-jvm) >> >> Dev environment >> We recommend checking out both projects and integrating them in a >> multi-project structure in your IDE. >> >> StreamPipes relies on a few external services, e.g. Consul, Couchdb, >> Kafka. To easify the development please use the StreamPipes CLI >> option to start your dev environment [3]. >> >> Here, there are various environments configurations depending on >> wether you want to develop: >> >> - core only -> backend >> - extensions only -> pipeline-element >> - core + extensions -> basic >> >> Go through the provided README file to see how to use the CLI. >> Briefly, it would be something like this for starting a docker dev >> environment with necessary services: >> >> ./streampipes env --set basic >> ./streampipes pull >> ./streampipes up -d >> >> In your IDE, set the the env variables for the services accordingly >> either using the provided run configs for IntelliJ or adding env >> variables for the StreamPipes to e.g. Eclipse by using the IntelliJ >> run configs as a template. >> >> We store this configs in Consul and do currently not overwrite them >> once a service has registered. >> You can either manually delete them from the Consul UI >> (localhost:8500) or simply start from a clean setup if you have a >> misconfiguration: >> >> ./streampipes clean -v >> >> NOTE! >> If non variables are present, all services use default variables that >> represent the prod settings when starting the containerized >> environment. Thus it is important to set these variables according to >> your setup. >> >> Networking in dev setup local machine <—> Docker This mixed setup: >> dev in IDE + Dockerized environment causes some network problems >> since use Consul (running in a container) to conduct service health >> checks to a service running on your local machine (in your IDE). >> Thus, we need an IP/DNS that is reachable/resolvable from inside the >> Docker container. >> >> Here it gets a little tricky as Docker currently does not provide a >> common solution for all OS. >> For instance, for the streampipes-pipeline-elements-all-jvm you will >> see in the run config that it sets an env variable as following [4]: >> >> <env name="SP_HOST" value="host.docker.internal" /> >> >> Here, SP_HOST refers to the IP/DNS of the service endpoint. >> Specifically, host.docker.internal is a Docker resolvable address on >> MacOS and Windows [5] >> >> For Linux: >> Whenever you see and env variable that uses „host.docker.internal“ as >> a value, you need to add the IP of you local machine >> >> —— >> >> Regarding your questions: >> >> 1. I was trying to start the backend module, which started fine, but >> when executing from UI, it is not able to connect to consul and mq >> due to host not found. I think I have not set the env variables >> correctly. I tried setting them in eclipse, but the host values are >> being taken from the consul key value store. How to change that to localhost? >> >> Related due to missing env variables when starting the backend main class. >> I assume, it’s trying to connect to something like consul:8500 >> instead of localhost:8500. >> Try with a clean setup, i.e., remove all docker volumes using the >> clean command above and setting the variables beforehand. >> >> >> 2. There are instructions to setup master container on the >> confluence, >> >> but >> >> I could not find its main class. >> >> This is most likely outdated, as explained above we recently >> integrated connect master into the backend. This is why you didn’t >> find a main class for the connect master. >> >> >> 3. Out of all the containers in docker, I am not sure of the function >> and code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to >> the backend api on http://backend:8030 by default. How do I change it >> to localhost? >> >> For the first part, I assume with function and code you mean where to >> find it? >> As said, we provide various bundles for pipeline elements. >> >> pipeline-element-all-jvm bundles all pipeline element modules of the >> extensions projects that are written for the plain Java wrapper (jvm) >> [6] (see the main class). >> >> connect-worker bundles all adapters [7] (see the main class). >> >> For the second part, this is mostly related again to missing env >> variables when starting the service. „backend“ is the default name of >> the docker service in prod. >> Take a look >> at the run configs for the env variables [8]. There you’ll find the >> following entry: >> >> <env name="SP_BACKEND_HOST" value="localhost" /> >> >> These env variables should be set in your IDE when starting your service. >> >> >> Hope this guide was helpful. >> >> We are currently restructuring the documentation, thus in the future >> we will add description as above right in the documentation. >> >> >> Patrick >> >> >> [1] >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon >> figurations >> < >> >> https://github.com/apache/incubator-streampipes/tree/dev/.idea/runCon >> figurations >> >> >> [2] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/. >> idea/runConfigurations >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/. >> idea/runConfigurations >> > >> >> [3] >> https://github.com/apache/incubator-streampipes-installer/tree/dev/cl >> i >> <https://github.com/apache/incubator-streampipes-installer/tree/dev/c >> li> >> [4] >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon >> figurations/all_pipeline_elements_jvm.xml >> < >> >> https://github.com/apache/incubator-streampipes/blob/dev/.idea/runCon >> figurations/all_pipeline_elements_jvm.xml >> > >> >> [5] https://docs.docker.com/docker-for-windows/networking/ < >> https://docs.docker.com/docker-for-windows/networking/> >> [6] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-pipeline-elements-all-jvm >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-pipeline-elements-all-jvm >> > >> >> [7] >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-connect-adapters >> < >> >> https://github.com/apache/incubator-streampipes-extensions/tree/dev/s >> treampipes-connect-adapters >> > >> >> [8] >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/. >> idea/runConfigurations/connect_adapters.xml >> < >> >> https://github.com/apache/incubator-streampipes-extensions/blob/dev/. >> idea/runConfigurations/connect_adapters.xml >> > >> >> >> Am 26.02.2021 um 15:50 schrieb Vineet Tiwari <[email protected]>: >> >> Hello, >> I have gone through some of the materials given above (thanks for >> that) >> >> and >> >> I have got a good idea on the high level architecture and design of >> the application. >> >> I started to set up streampipes on my local machine as well. >> I have the below queries: >> 1. I was trying to start the backend module, which started fine, but >> when executing from UI, it is not able to connect to consul and mq >> due to host not found. I think I have not set the env variables >> correctly. I tried setting them in eclipse, but the host values are >> being taken from the consul key value store. How to change that to localhost? >> 2. There are instructions to setup master container on the >> confluence, >> >> but >> >> I could not find its main class. >> 3. Out of all the containers in docker, I am not sure of the function >> and code inside these two containers - pipeline-element-all and >> >> connect-worker. >> >> Also, once started, the connect-worker keeps on trying to connect to >> the backend api on http://backend:8030 by default. How do I change it >> to localhost? >> >> Thanks, >> Vineet Tiwari >> >> On Wed, Feb 24, 2021 at 2:45 AM Dominik Riemer <[email protected]> >> >> wrote: >> >> >> Hi Vineet, >> >> welcome! >> If you have any questions on how to get started, we'll be glad to help. >> >> Doominik >> >> On 2021/02/23 06:41:06, Vineet Tiwari <[email protected]> wrote: >> >> Hello, >> My name is Vineet Tiwari and I am working as a full time software >> developer. I am pretty good at Java and javascript (mostly with >> >> ReactJs) >> >> languages along with basic C++ and Python. >> I have a total of three and a half years of experience working as a >> SDE >> >> on >> >> several projects primarily with JAVA programming language. >> >> I do not have any prior open source contributions. I have just >> started >> >> to >> >> explore open-source projects and I found this project very >> >> interesting. I >> >> have gone through the website as well as the confluence page and I >> feel that I can surely contribute well to this project with my skills >> and capabilities. >> >> I have subscribed to the dev and commits mailing lists and I look >> >> forward >> >> to hearing from you about how I can contribute to the project. >> >> Thanks and regards >> Vineet Tiwari >> >> >>
