Hi Fede, ok I think I understand the problem now. The compose component expects two data streams. Data Sets are replayed when the pipeline is started. Thereby the data is played "as fast as possible". That means when the data from the other data stream comes, all the data from the data set has already been played. So, they cannot be processed in the Compose component.
Since we have several problems with datasets, we had a discussion [1] about it. We decided to remove them in the future releases and concentrate on streams. Cheers, Philipp [1] https://github.com/apache/streampipes/discussions/1115 Von: Rullo, Federico <[email protected]> Datum: Dienstag, 9. Mai 2023 um 09:18 An: [email protected] <[email protected]> Betreff: RE: Compose Issue Hello Philipp, Thank you for your reply and please ignore the previous email as it was not complete. Yes, we are using the adapter with the schema (date_unix_ts, value) as input to the signal monitoring processor and we have used a fixed output strategy to name the timestamp ‘date_unix_ts’ as 'timestamp'. Moreover, the conversion in our onEvent method was well done. Therefore, it is not the reason for the problems with the compose component; also, because when we connect directly our adapters to the compose component we experienced the same issue. On further testing, we noticed that this could be related to the type of data source (set or stream) we are using: By using data source from a file set adapter, we get the same problem. But by using a file stream adapter with the same data, the compose component responds better. So, since our HTTP stream-based data source adapter retrieves a set of data at each polling and passes it to a processor like a file set, it seems that the compose component is not able to handle the sets of data we give to it. Thank you in advance, Best, Fede Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected] www.gft.com/it<http://www.gft.com/it> https://blog.gft.com https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]] From: Philipp Zehnder <[email protected]> Sent: venerdì 5 maggio 2023 16:34 To: [email protected] Subject: AW: Compose Issue [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Fede, so I assume you are using the adapter with the schema (date_unix_ts, value) as input to the signal monitoring processor, right? It looks like you used a fixed output strategy and named the timestamp ('timestamp'). In the original stream the timestamp is named ('date_unix_ts'), did you convert this value accordingly in your onEvent method, or could this be the reason for the problems you have with the compose component? Cheers, Philipp ………………………………………………… Philipp Zehnder Bytefabrik.AI GmbH Haid-und-Neu-Str. 10–14 76131 Karlsruhe Tel.: +49 721 9654-805 [email protected]<mailto:[email protected]> www.bytefabrik.ai <https://www.bytefabrik.ai/> | www.twitter.com/bytefabrik<https://www.bytefabrik.ai/> ………………………………………………… Bytefabrik.AI GmbH Sitz der Gesellschaft: Karlsruhe Registergericht: Amtsgericht Mannheim, HRB 741226 Geschäftsführer: Dr.-Ing. Dominik Riemer, Dr.-Ing. Philipp Zehnder ………………………………………………… Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Dienstag, 2. Mai 2023 um 14:23 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: RE: Compose Issue Hi Philipp, Please find attached some screenshots of our schema. Best, Fede. Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<http://www.gft.com/it> https://blog.gft.com<https://blog.gft.com/> https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]] From: Philipp Zehnder <[email protected]<mailto:[email protected]>> Sent: martedì 2 maggio 2023 11:10 To: [email protected]<mailto:[email protected]> Subject: AW: Compose Issue [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Federico, what is the output strategy of your processors? Cheers, Philipp Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Mittwoch, 26. April 2023 um 09:46 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: RE: Compose Issue Hi Philipp, We also tried with a vanilla version of the 0.91.0 stable and had no luck. We are creating our own, Data Sources, yes, but the problem we have is only with the Compose and the Synchronize Streams, which shows only the stream of data of which we selected the timestamp but seems to be working if we create a datasource using the machine simulator adapter. On the other hand, most of the default components work with our “custom” data sources. To give you more context our data is composed of a timestamp field, and a value field. The custom adapter that creates this data sources is based on the Http Stream adapter. Best, Federico. Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<http://www.gft.com/it> https://blog.gft.com<https://blog.gft.com/> https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]] From: Philipp Zehnder <[email protected]<mailto:[email protected]>> Sent: martedì 25 aprile 2023 16:01 To: [email protected]<mailto:[email protected]> Subject: AW: Compose Issue [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Federico, sorry as far as I know the problem this is not a known issue. Maybe you can try it with the newest version 0.91.0. If I understand it correctly you implemented your own components that produce the input for the compose processor, right? Maybe you can check if there is a problem with the output events from your component. Cheers, Philipp Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Mittwoch, 19. April 2023 um 16:35 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: Compose Issue Hello, When we use the "compose" component to aggregate data, it does not output all data from the stream where the timestamp was not selected but returns all data from the stream with the selected timestamp. We also tried to aggregate our data with the "synchronize two streams" component, but it does not output any our data even though it consumes the data. Moreover, when we use these components with two streams ad hoc from the machine data simulator, they work well. Do you know if this is a know issue? Can you give us more information about it? We are currently running on the version 0.70.0, because of an issue with version 0.90.0. Best, Federico. Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<http://www.gft.com/it> https://blog.gft.com<https://blog.gft.com/> https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]
