Doing the same than sax/jaxb with jsonp sounds very feasible too - that's the impl i had in mind.
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le ven. 23 oct. 2020 à 18:05, Claus Ibsen <claus.ib...@gmail.com> a écrit : > Ah there is a nice jsonsurfer project that adds streaming support. > I have created JIRA: https://issues.apache.org/jira/browse/CAMEL-15746 > > > On Fri, Oct 23, 2020 at 5:59 PM Claus Ibsen <claus.ib...@gmail.com> wrote: > > > > Hi > > > > Just a note about splitting big xml files. Then there is camel-stax > > and also the xtokenizer language you can use in splitter with > > streaming. > > We have some blogs and articles about that. > > > > For json then the jsonpath project have a ticket about adding support > > for streaming, but the maintainers of that project is no very active > > so I doubt they get around working on that. A PR was in the works that > > is stalled: https://github.com/json-path/JsonPath/pull/93 > > > > But from camel you can split anything in streaming mode by passing in > > something that can be iterated - then its splitted in chunks. > > > > If your json file is structured the same way - then you could write an > > iterator that reads the content and emit chunks when you find where > > each "array" ends. > > > > We may consider adding an jtokenizer (json tokenizer) to camel that > > works a bit the same. > > > > On Fri, Oct 23, 2020 at 4:39 PM Romain Manni-Bucau > > <rmannibu...@gmail.com> wrote: > > > > > > Hi, > > > > > > From my experience - got the same issue with xml years ago now, the > > > simplest is to do a custom component (like jsonstreaming:....) and > handle > > > the split in this component. > > > It will likely require a jsonpointer to find an array to emit array > item > > > one by one to next processor. > > > At the end the route could look like something like: > > > > > > from("jsonstreaming:/path/file.json?pointer=/items") > > > .to("bean:processUser"); > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://twitter.com/rmannibucau> | Blog > > > <https://rmannibucau.metawerx.net/> | Old Blog > > > <http://rmannibucau.wordpress.com> | Github < > https://github.com/rmannibucau> | > > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > > > > > > > Le ven. 23 oct. 2020 à 16:16, Mantas Gridinas <mgridi...@gmail.com> a > > > écrit : > > > > > > > From personal experience any activities related to file component > tend to > > > > try to load entire file into memory. You could perhaps fiddle around > by > > > > converting it to an input stream but then you get into an issue of > making > > > > sure that you don't read an entire file into memory before actually > > > > converting it. > > > > > > > > I'd suggest avoiding using camel here at all for the sake of > retaining fine > > > > grained control over streaming process. At most you could wrap it in > a > > > > processor and be done with it. Otherwise you'll start programming > with > > > > routes, which is something you'd want to avoid. > > > > > > > > > > > > On Fri, Oct 23, 2020, 17:05 Site Register <site.regis...@ymail.com > > > > .invalid> > > > > wrote: > > > > > > > > > Hi Camel Users, > > > > > I have a 4G json array file need to load into database. How can I > > > > leverage > > > > > Camel to stream the file and split into json? > > > > > I had tried to use "stream:file" but it was reading line by line > not > > > > split > > > > > into json. > > > > > I leveraged gson streaming to read and insert into the database in > java > > > > > application which took about 3 minutes. However, I would like to > check if > > > > > there is a way to leverage camel pipeline for the same purpose. > > > > > Thank you, > > > > > > > > > > > > -- > > Claus Ibsen > > ----------------- > > http://davsclaus.com @davsclaus > > Camel in Action 2: https://www.manning.com/ibsen2 > > > > -- > Claus Ibsen > ----------------- > http://davsclaus.com @davsclaus > Camel in Action 2: https://www.manning.com/ibsen2 >