Hi Charles, I’ll port this over to the mailing list as maybe other users run into similar problems.
First of all, congrats on your ISS pipeline :) I still find this a cool initial demo pipeline with real world open data. Regarding the problem, as of now, without looking into the logs I’m not quiet sure what might cause this as you stated that all containers are up and running. Restarting could be an option, but I suggest to gracefully stop the pipelines from the UI before performing down/up command with docker-compose. We just recently added this in the current dev version to handle ungraceful shutdowns where pipelines are flagged accordingly. Maybe the others have some ideas? In the meantime, maybe you can spot anything suspicious in the backend logs: docker-compose logs backend Best Patrick > Anfang der weitergeleiteten Nachricht: > > Von: charlesearl <[email protected]> > Betreff: Aw: Missing Elements > Datum: 19. März 2021 um 11:33:08 MEZ > An: Patrick Wiener <[email protected]> > > HI Patrick > > Thank you for your helpful email noting critical settings. > > My ISS Location pipeline, with just the IL, RG, Speed Calculator, Dashboard > Sink connected does work under the full version. Currently flying down the > Pacific coast of Peru. > > The ‘Cloud’ view still shows only the spinning “Searching for ….. please > wait…”. > Manage Endpoints gives “Existing Endpoints” without specification. > > Pipeline element configuration shows 4 containers, all status Green. They > are: Flink, JVM, VehicleSimulator, Watertank Simulator. > > Data Marketplace shows 27 Data Stream and 6 Data Set elements. > > Do you think I should restart by down / up the docker-compose in terminal, If > so would anything be saved? > > I do hope these few markers are helpful as I’m sure you shall find the > solution. > > Best wishes > Charles > > > >> On 18 Mar 2021, at 18:14, Patrick Wiener <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Charles, >> >> Just to make sure: does the actual ISS pipeline now work with the full >> version? >> >> Regarding the pipeline element view it should look similar to the one >> attached. Technically, what we do here is (1) query consul for healthy >> pipeline element service container (b) query each pipeline element service >> endpoint to provide its description. >> >> You can check out the configuration view (settings icon on the left) under >> „Pipeline Element Configuration“ you should see a „green“ status of all >> healthy pipeline elements services container. If this is case all seems to >> be running well. >> >> Maybe one question: Does it eventually render something? You should have >> something like >100 pipeline elements and we have not optimized this part of >> StreamPipes as we are about to internally >> replace a graph storage using JSON-LD with plain JSON. Hitting the graph DB >> and/or deserializing/serializing the JSON-LD data format takes quiet long. >> Thus, maybe it takes a while … but not like >> minutes or so. >> >> Best >> Patrick >> >> <PastedGraphic-1.png> >> >>> Am 18.03.2021 um 19:00 schrieb charlesearl <[email protected] >>> <mailto:[email protected]>>: >>> >>> Hi Patrick >>> >>> I don’t wish to interrupt your evening but I must thank you for the useful >>> hint to display data from the ISS source on the dashboard. It proved very >>> encouraging. I installed the full version beforehand which seems to have >>> worked. I expanded the pipeline with the RG and Speed calculator elements >>> and displayed the map, speed and nearest city in the dashboard. At this >>> point I tried to download other elements through the “install pipeline >>> elements” facility but get “searching for available pipeline elements, >>> please wait…” >>> >>> It doesn’t seem possible to install at present and this may well be due to >>> some configuration I’ve missed. >>> You have been very kind in getting me progressing with the installation and >>> display features and I’m really thankful. There’s no hurry to go further >>> but if you can easily see a missing link then I should welcome your advice. >>> >>> Best wishes >>> Charles >>> >>> >>>> On 17 Mar 2021, at 20:51, Patrick Wiener <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi Charles, >>>> >>>> perfect. I just was not quiet sure and hopefully didn’t bother you with >>>> the second mail. >>>> >>>> If you won’t get it to work I’ll be more than happy to help. As a minor >>>> hint in the beginning you can >>>> always try with the most simplistic pipeline which is simply connect the >>>> source with the dashboard >>>> and after starting, create a new visualization using the „raw“ widget. >>>> >>>> This simply logs the raw events. >>>> >>>> In the future we plan to incorporate such a feature in the early beginning >>>> of the pipeline modeling phase >>>> to better understand current event schemas and view actual events in the >>>> middle of the pipeline assembly. >>>> But for now, this is the best bet to see if the system itself is properly >>>> working. >>>> >>>> Weather here in Karlsruhe is kind of April-like already :) from warm to >>>> cold, to sunny, rainy and actual hail >>>> today :) >>>> >>>> Best >>>> Patrick >>>> >>>>> Am 17.03.2021 um 21:42 schrieb charlesearl <[email protected] >>>>> <mailto:[email protected]>>: >>>>> >>>>> Hi Patrick >>>>> >>>>> Many thanks for your replies to my request. I did receive the first >>>>> promptly and followed your advice on the ISS data source. It does give >>>>> updates on timestamp and lat/long though I have yet to get the output on >>>>> the dashboard. Possibly, I have configured the data processors >>>>> incorrectly. Therefore, I thought to go over your tutorials to try to >>>>> fix the output. >>>>> >>>>> I have yet to upgrade to the full version but shall do so. >>>>> >>>>> I hope you have enjoyed equally fine Spring weather in Karlsruhe. >>>>> >>>>> Best wishes >>>>> Charles >>>>> >>>>> >>>>>> On 17 Mar 2021, at 20:01, Patrick Wiener <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Hi Charles, >>>>>> >>>>>> idk wether you got the mail via the users mailing list - thus I’ll >>>>>> forward it again. >>>>>> I just tested the public ISS API endpoint - it should deliver data, at >>>>>> least it did right now. >>>>>> >>>>>> Patrick >>>>>> >>>>>> >>>>>>> Anfang der weitergeleiteten Nachricht: >>>>>>> >>>>>>> Von: Patrick Wiener <[email protected] <mailto:[email protected]>> >>>>>>> Betreff: Aw: Missing Elements >>>>>>> Datum: 15. März 2021 um 23:25:40 MEZ >>>>>>> An: <[email protected] <mailto:[email protected]>> >>>>>>> Antwort an: <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> >>>>>>> Hi Charles, >>>>>>> >>>>>>> welcome to the StreamPipes mailing list and happy to share your >>>>>>> experience with us. >>>>>>> >>>>>>> Of course, we can further help you regarding your problems. >>>>>>> >>>>>>> For the one regarding the tutorial: the missing data processor is now >>>>>>> part of the „full“ or „big“ version of StreamPipes [1] >>>>>>> as this pipeline element (data processor) is written using our Apache >>>>>>> Flink wrapper. Thus, in the standard getting started >>>>>>> we now guide the user towards using the „lite“ version as we >>>>>>> experienced some resource issues when starting full fletched >>>>>>> version including external frameworks such as Flink in the case of the >>>>>>> full version. Unfortunately, we have not yet updated >>>>>>> the documentation - so apologize for that. >>>>>>> >>>>>>> With your system you should be able to run the full version without any >>>>>>> issues and complete the second tutorial. Therefore, >>>>>>> just clean the old setup and start the corresponding >>>>>>> docker-compose.full.yml. >>>>>>> >>>>>>> For your second problem, as of now I cannot really say much about it. >>>>>>> Did you experience any issues when creating the ISS >>>>>>> adapter? Could you see gathered data in the live preview (Pipeline >>>>>>> Editor -> Data Streams -> Element, then click the „?“ and >>>>>>> go to „Values“ -> you should see „Gathering data for live preview …“ >>>>>>> followed by live data from the ISS adapter). >>>>>>> >>>>>>> Don’t hesitate to further reach out to us :) >>>>>>> >>>>>>> Patrick >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/apache/incubator-streampipes-installer/blob/dev/compose/docker-compose.full.yml >>>>>>> >>>>>>> <https://github.com/apache/incubator-streampipes-installer/blob/dev/compose/docker-compose.full.yml> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Am 15.03.2021 um 23:03 schrieb Charles Earl >>>>>>>> <[email protected] <mailto:[email protected]>>: >>>>>>>> >>>>>>>> Hello >>>>>>>> >>>>>>>> I am new to Apache Stream Pipes. Following the tutorial I >>>>>>>> successfully built and used the first example, the flow valve >>>>>>>> simulation. I could not build the the second example because the >>>>>>>> 'Aggregator' data processor element was missing. I tried to Pull it >>>>>>>> using the terminal command but could not install it. I have >>>>>>>> subscribed as a user. >>>>>>>> >>>>>>>> The ISS Location Stream Pipe, exampled in the recent edition of Linux >>>>>>>> Magazine connected fully but my Dashboard failed to show the data >>>>>>>> output. I am still learning but find your concept very interesting >>>>>>>> and worth the challenge. >>>>>>>> >>>>>>>> If you could help on these points I shall be very thankful. The >>>>>>>> system I am using is Ubuntu 20.04 running on a desktop AMD Phenom with >>>>>>>> 16GB Ram. >>>>>>>> >>>>>>>> With thanks and best wishes >>>>>>>> >>>>>>>> Charles Earl >>>>>>>> >>>>>>>> Email: [email protected] >>>>>>>> <mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
