MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-465244820
@markap14 @bbende would you rather I refactor the JsonTreeReader to use the
JsonSurfer library or keep this one separate?
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447923122
I don't think there is because the Jayway jsonpath library reads the entire
InputStream into memory before doing anything with it. JsonSurfer uses
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447902019
@markap14 I can rename it to `StreamingJsonPathReader`. Does that work?
This is
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447891669
I was vaguely tracking the SplitJson issue, so I knew about the issue before
even starting on the work.
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447889903
@ottobackwards we definitely should refactor the readers at some point to
use common code. Problem I ran into is that the existing ones are using
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447879070
> @MikeThomsen are you saying that your 25GB JSON file got an OOM (or some
error) when using ConvertRecord with JsonTreeReader?
Yeah, in
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447875162
> I don't see how you avoid reading that all into memory because you have to
return a Record instance from the reader with all that data in it.
MikeThomsen commented on issue #3222: NIFI-5900 Added StreamingJsonReader.
URL: https://github.com/apache/nifi/pull/3222#issuecomment-447833164
@markap14 @mattyb149 @ijokarumawak @zenfenan @ottobackwards
We got a file format that looks roughly like this:
```
{
