Hi Federico, can you also try to delete all the volumes on a test StreamPipes instance (e.g. on a test VM). Maybe there are some ‘old’ descriptions stored in the database. Or did you already try it on a clean installation?
Philipp Von: Rullo, Federico <[email protected]> Datum: Dienstag, 11. Oktober 2022 um 16:52 An: [email protected] <[email protected]> Betreff: RE: Pipeline elements not showing and Datalake not receiving data Hi Philipp, To build everything, I log in with my docker account, then I do mvn clean package, then go to streampipes-pipeline-elements-all-jvm and do docker build -t with a slight different name, and then I push it to the registry. Finally, I set the environment to full and start streampipes. Every time I make some changes I clean all the images and containers on docker, so that it is empty every time. Best, Federico Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected] www.gft.com/it<http://www.gft.com/it> https://blog.gft.com https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso> From: Philipp Zehnder <[email protected]> Sent: martedì 11 ottobre 2022 16:48 To: [email protected] Subject: AW: Pipeline elements not showing and Datalake not receiving data [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Federico, thanks, providing the details. When you run it in the development environment (within your IDE) everything works fine, but when you run them in docker you get the errors you explained, correct? How are you building the docker images? Is it possible that you start an “old” container that does not include the newest version of you processor implementation? Usually, you should get the same behavior no matter if the processors run within the IDE or within a docker container. To me it sounds that there is a problem with the build pipeline. Philipp Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Montag, 10. Oktober 2022 um 08:59 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: RE: Pipeline elements not showing and Datalake not receiving data Hi Philipp, So, I currently have 4 processors into the SP installation, but one of them is not showing, and the other two are not sending any data to the datalake. If I run them through the IDE they work fine, but if I run them inside the full environment of SP, they have these issues. I also noticed that even if I clean up all the containers and images, when I make changes sometimes, they are not applied, for example if I change the name of the output of a processor, when I run the real-time preview, I still see the old name. This are some of the error messages I get in the logs: SP-log.log:pipeline-elements-all-jvm_1 | 08:04:13.557 SP [XNIO-1 task-2] ERROR o.a.s.c.a.InvocablePipelineElementResource - Could not invoke pipeline element Data Lake due to the following error: 409 - SP-log.log:backend_1 | 08:08:06.716 SP [http-nio-8030-exec-6] ERROR o.a.c.c.C.[.[.[.[.a.s.b.StreamPipesResourceConfig] - Servlet.service() for servlet [org.apache.streampipes.backend.StreamPipesResourceConfig] in context with path [/streampipes-backend] threw exception [java.lang.NullPointerException] with root cause. Let me know if it is much clearer now, in the meantime I will continue checking the logs for issues regarding the datalake. Best, Federico Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916135730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6ETbFt3VTD%2F0slmqtA%2F4bivx2%2F%2BcAv0PR%2FABf%2BMflb8%3D&reserved=0> https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916135730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QF%2BVmxmVoOvxX0JckzksqV4qZJkpaKoQUSQokfYeSC0%3D&reserved=0> https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916135730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ec%2FSfqz0CW%2FQZPTLUNiUQfVmxsr%2FRNK%2B46obZ%2FGFpzY%3D&reserved=0> www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916135730%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=at8J6%2FRIWk%2ByX1KAZ4nKiHgMoBGvKlNwAjx5jI%2Bjo2k%3D&reserved=0> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dl%2BEt%2FSjLmJvdMEDXYLghi0as6Mz%2FXt%2BIaGjb47bTqs%3D&reserved=0> From: Philipp Zehnder <[email protected]<mailto:[email protected]>> Sent: venerdì 7 ottobre 2022 13:39 To: [email protected]<mailto:[email protected]> Subject: AW: Pipeline elements not showing and Datalake not receiving data [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Frederico, do I understand it correctly two of the new processors show up, but two are not displayed in “Install Pipeline Elements”? If this is the case, can you please check the identifiers of these processors. Maybe they are the same for the new processors. Regarding the data that is not written to the data lake, is there an error message in the logs? Cheers, Philipp Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Mittwoch, 5. Oktober 2022 um 17:54 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: Pipeline elements not showing and Datalake not receiving data Hello, I have developed a new custom processor and added them, the same way I did for the previous ones, arriving to have four custom processors on SP, but it doesn’t show on SP when I start it, while also two of the custom components that were already on it, works since using the real-time data view shows the results but do not send any data to the datalake. Do you have any idea what might be the problem? Can we maybe schedule a call? Thanks in advance for the reply, Best, Federico. Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tIIiayU98EIMHX8%2B2tYfnFz6mYbaHtwndnBS%2FeIxkfw%3D&reserved=0> https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xKL8kaFfcd%2BdbJCqw5X2WivNNXZwNa7Qf9gP%2BduUiFw%3D&reserved=0> https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WkDNsdw5CtIune781vKf1%2F7bG015XEs39UPGwyR6V2I%3D&reserved=0> www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1tyLixSMtsoLK742X3X4kDLp7%2B7giOTvkfVBKYFKXxA%3D&reserved=0> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfd4b072c76f041ab1ac108daab979cf8%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638010964916291954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dl%2BEt%2FSjLmJvdMEDXYLghi0as6Mz%2FXt%2BIaGjb47bTqs%3D&reserved=0>
