Re: NiFi for Easy and Efficient Data Ingestion from Different Sources into Delta Lake

2020-03-29 Thread Martin Ebert
Hi Joe, since it would actually create a huge added value for us (and certainly for many people in the community) if everything worked out the way we wanted it to, I will actively pursue the topic. We have the contacts to Databricks and will ask them what possibilities there are here. I think

Re: Reading the incoming flowfile "twice"

2020-03-29 Thread Russell Bateman
No, this test file is tiny. The real thing is usually megabytes in size. On Sun, Mar 29, 2020 at 3:15 AM Mike Thomsen wrote: > If these files are only a few MB at the most, you can also just export them > to a ByteArrayOutputStream. Just a thought. > > On Sun, Mar 29, 2020 at 12:16 AM Russell

Re: Reading the incoming flowfile "twice"

2020-03-29 Thread Joe Witt
Russell I recommend writing very simple code that does two successive read/write operations on basic data so you can make sure the api work/as expected. Then add the xml bits. Thanks On Sun, Mar 29, 2020 at 5:15 AM Mike Thomsen wrote: > If these files are only a few MB at the most, you can

Re: Reading the incoming flowfile "twice"

2020-03-29 Thread Mike Thomsen
If these files are only a few MB at the most, you can also just export them to a ByteArrayOutputStream. Just a thought. On Sun, Mar 29, 2020 at 12:16 AM Russell Bateman wrote: > Joe and Mike, > > Sadly, I was not able to get very far on this. It seems that the extend > to which I copy the first