Hi Federico, thanks a lot for providing the sample file.
I loaded it into a local StreamPipes instance and used the file adapters to read it. As the format I had to choose “Array Field” with the “Key: data”, because the data is in the array with the key “data”, correct? I selected the field “date_unix_ts” for the timestamp and I was able to write the data into the data lake and visualize it in a data view. For that I used the “File Set” adapter (this adapter replays the file only once). This is a good way to load data into the data lake. If you want to work with the data in a pipeline, I would recommend to use the “File Stream” adapter. This will play back the original data in an infinite loop, but you must remember that the original timestamps will also be used as the original replay speed. I hope this helps. Please let us know if you have any other problems. Cheers, Philipp Von: Rullo, Federico <[email protected]> Datum: Donnerstag, 24. November 2022 um 10:09 An: [email protected] <[email protected]> Betreff: RE: Cannot create new datastream from JSON file Hi Philipp, Here is a file sample. F. Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected] www.gft.com/it<http://www.gft.com/it> https://blog.gft.com https://www.linkedin.com/company/gft-group/ www.twitter.com/gft_it <http://www.twitter.com/gft_it%20> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso> From: Philipp Zehnder <[email protected]> Sent: mercoledì 23 novembre 2022 17:58 To: [email protected] Subject: AW: Cannot create new datastream from JSON file [EXTERNAL EMAIL] This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe. Hi Federico, can you provide a minimal sample json that is not working? Then I will test it locally to see what is going wrong. It is strange that the preview feature works with a different replay speed. The replay speed configuration should only be used once the adapter is started to reduce the time between the events. Cheers, Philipp Von: Rullo, Federico <[email protected]<mailto:[email protected]>> Datum: Montag, 21. November 2022 um 17:54 An: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Betreff: Cannot create new datastream from JSON file Hello, The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don’t see any data previewing when the adapter is created but if we put 50 we do get the preview. In both cases, no output is produced from the created adapters, and no errors is displayed in the logs. The steps we are taking to create the adapters are: 1. Upload the JSON file 2. Input 1 as replay speed 3. Select JSON and JSONArray as file format 4. Selecting the data under the “data” filed 5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion 6. Name the adapter and start it After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data. Finally, if we create a new pipeline using the newly created adapter, no data is passed. Hence, we see no data in the data explorer and inside the influxdb itself. We had a similar issue with the preview’s version, do you know if there is anything we are doing wrong or if there is some problem handling jsons? The version we are currently using is the 0.70.0 set on the pipeline-element environment. Thanks in advance, Best Regards, Federico Rullo Federico Rullo _________________________________________________________________ GFT Italia S.r.l. Via A. Rimassa, 51/2 16129 Genova, Italy T +39 025 8158169 [email protected]<mailto:[email protected]> www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183279684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wTGCGWNuNVE7pto%2BS6jcmBCZHeHVpppFl0TW2mo3%2Bqs%3D&reserved=0> https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt6m0TtoeT5bSmK4QzEDHgYSbyAm6s%2F1UtFVCSEXnC4%3D&reserved=0> https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U61BOSD8mVYtOq%2BzkOhgicfHtS2b9nnQxaQmecT7c%2Fs%3D&reserved=0> www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CPiRwQNwbTLZ4kvrEtZ0KA4iI7onqVr0sgTfQlH7is4%3D&reserved=0> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy) VAT Code and number of registration with the Companies' Register of Milan: 00819200478 Corporate capital: Euro 1,000,000 fully paid-in Company subject to the direction and coordination activity of GFT Technologies SE Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company. _________________________________________________________________ [cid:[email protected]]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KZcQTHhxXJsNXEcQdATMHHp1UPwtwQ%2BMFif%2BeI7R3vo%3D&reserved=0>
