Re: Issue when extracting a large dataset into CSV files

2021-05-10 Thread Matt Burgess
Vibath, What is the "Fetch Size" property set to? It looks like PostgreSQL will load all results if Fetch Size is set to zero [1]. Try setting it to 1 or something like that, whatever doesn't use too much memory but doesn't slow down the performance too much. Regards, Matt [1] https://jdbc.p

Re: JoltTransformRecord producing IllegalTypeConversionException

2021-05-10 Thread Gregory M. Foreman
Jim: Thank you for the suggestions. I adjusted the schema to only null or string array and modified the incoming flowfile to cooperate and it worked. Greg > On May 6, 2021, at 3:57 PM, Jim Williams wrote: > > Greg - > > Maybe use the standard Jolt transformation processor (not the record or

Re: Issue when extracting a large dataset into CSV files

2021-05-10 Thread Vibhath Ileperuma
Hi Vijay, Eventhough that property is set to a postive value, same issue happens. Seems like NIFI first loads all the rows to RAM and write to multiple files when that property is set to a non-zero value. On the other hand, if this property is set to a very small value, a large no of flow files ca