On Wed, Jun 15, 2022 at 2:34 PM Rodrigo Ottero <[email protected]> wrote: > > TL;DR: > > Is there any way I could stream data to a huge json file in small chunks, but > adding/appending said data to one field only? > > I mean, this json file represents one single pojo object, and I need to keep > appending data to one of its fields.
Incremental output is indeed possible, but if output is not a sequence of elements, SequenceWriter is probably not the way to go. You can still use `ObjectMapper` (and `ObjectWriter`) `writeValue()` methods that take `JsonGenerator`, however. So usually you will need to write "outer" part of content straight using `JsonGenerator` (or CsvGenerator etc) methods, and then for serializing values within, use ObjectMapper. Or use `JsonGenerator`, depending on case. I hope this helps, -+ Tatu +- > > Thanks! > > (sent by smartphone / enviado pelo celular) > > On Wed, 15 Jun 2022, 22:20 Rodrigo Ottero, <[email protected]> wrote: >> >> Hi folks, >> >> Apologies in advance if I am not clear enough, happy to provide >> clarifications if required. >> >> I am having a hard time finding information about how to use SequenceWriter >> to output a custom POJO to a JSON file in an incremental way. >> >> I found Bozhidar's solution to incrementally output data as a JSON array >> (https://dzone.com/articles/writing-big-json-files-with-jackson); that is >> natively supported by SequenceWriter through the init(true) method. >> >> In my use case, we have a service that receives a SQL query and outputs it >> as a CSV file. >> >> Since some queries may produce millions of rows, this CSV is written >> incrementally as the result set is read, line by line, to prevent out of >> memory issues. >> >> I was asked to add the option of outputting a JSON file instead of a CSV >> file, but in the same incremental way. >> >> The gotcha is that I need to use a legacy JSON structure, something like: >> >> { >> "metadata": { >> "query": "SELECT * FROM ZZZ", >> "processingTime": 100, >> }, >> "result": { >> "columns": [ >> "Column1", >> "Column2", >> "..." >> ], >> "result": [ >> [ >> "abc", >> "def", >> "..." >> ], >> [ >> "foo", >> "bar", >> "..." >> ], >> ...millions of lines... >> ] >> } >> } >> >> The part in red is the part I want to incrementally write/update on the JSON >> file, while keeping the blue parts fixed. >> >> I was browsing the javadocs for SequenceWriter >> (https://developer.adobe.com/experience-manager/reference-materials/6-5/javadoc/com/fasterxml/jackson/databind/SequenceWriter.html) >> and saw that the constructor accepts a DefaultSerializerProvider and a >> JsonGenerator object, but I am not sure how I would use these resources to >> achieve my objective. >> >> Would you have a piece of documentation or an example of how I could use >> SequenceWriter to incrementally write the JSON file I need, based on a >> custom POJO? >> >> Or could you please tell me if I should be looking for another approach, in >> case SequenceWriter is not the answer for my use case? >> >> Many thanks, >> Best regards, >> Rodrigo >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "jackson-user" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/jackson-user/kXvRr6b7rg4/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jackson-user/01796d7e-5b37-4180-94ea-a7be908d4efan%40googlegroups.com. > > -- > You received this message because you are subscribed to the Google Groups > "jackson-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jackson-user/CAOmZcH%2BxxXQoSTw7cTMsnZ1eHtcQtmuMgbkYDxkWjYMHeoeH_Q%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "jackson-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/CAL4a10gTeFedMUvfX8tAgQWgK6CsNGA_NnbqzArquzk0zK564g%40mail.gmail.com.
