MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader. URL: https://github.com/apache/nifi/pull/3222#issuecomment-447875162 > I don't see how you avoid reading that all into memory because you have to return a Record instance from the reader with all that data in it. Our use case has someone throwing a 10GB file with that structure, so JsonTreeReader would have to have a schema reference to the massive array and load it all at once. What this does it uses a streaing JsonPath parser to go directly to the array and pull each element one by one and expose them to the reader. As I understand it, JsonTreeReader can only handle such a large file if it contains only an array of elements or the elements stacked on each other. If the NiFi user has to drill into the document at all, they're out of luck.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
