Doing the same than sax/jaxb with jsonp sounds very feasible too - that's
the impl i had in mind.

Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> |  Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>


Le ven. 23 oct. 2020 à 18:05, Claus Ibsen <claus.ib...@gmail.com> a écrit :

> Ah there is a nice jsonsurfer project that adds streaming support.
> I have created JIRA: https://issues.apache.org/jira/browse/CAMEL-15746
>
>
> On Fri, Oct 23, 2020 at 5:59 PM Claus Ibsen <claus.ib...@gmail.com> wrote:
> >
> > Hi
> >
> > Just a note about splitting big xml files. Then there is camel-stax
> > and also the xtokenizer language you can use in splitter with
> > streaming.
> > We have some blogs and articles about that.
> >
> > For json then the jsonpath project have a ticket about adding support
> > for streaming, but the maintainers of that project is no very active
> > so I doubt they get around working on that. A PR was in the works that
> > is stalled: https://github.com/json-path/JsonPath/pull/93
> >
> > But from camel you can split anything in streaming mode by passing in
> > something that can be iterated - then its splitted in chunks.
> >
> > If your json file is structured the same way - then you could write an
> > iterator that reads the content and emit chunks when you find where
> > each "array" ends.
> >
> > We may consider adding an jtokenizer (json tokenizer) to camel that
> > works a bit the same.
> >
> > On Fri, Oct 23, 2020 at 4:39 PM Romain Manni-Bucau
> > <rmannibu...@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > From my experience - got the same issue with xml years ago now, the
> > > simplest is to do a custom component (like jsonstreaming:....) and
> handle
> > > the split in this component.
> > > It will likely require a jsonpointer to find an array to emit array
> item
> > > one by one to next processor.
> > > At the end the route could look like something like:
> > >
> > > from("jsonstreaming:/path/file.json?pointer=/items")
> > >   .to("bean:processUser");
> > >
> > > Romain Manni-Bucau
> > > @rmannibucau <https://twitter.com/rmannibucau> |  Blog
> > > <https://rmannibucau.metawerx.net/> | Old Blog
> > > <http://rmannibucau.wordpress.com> | Github <
> https://github.com/rmannibucau> |
> > > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> > >
> > >
> > > Le ven. 23 oct. 2020 à 16:16, Mantas Gridinas <mgridi...@gmail.com> a
> > > écrit :
> > >
> > > > From personal experience any activities related to file component
> tend to
> > > > try to load entire file into memory. You could perhaps fiddle around
> by
> > > > converting it to an input stream but then you get into an issue of
> making
> > > > sure that you don't read an entire file into memory before actually
> > > > converting it.
> > > >
> > > > I'd suggest avoiding using camel here at all for the sake of
> retaining fine
> > > > grained control over streaming process. At most you could wrap it in
> a
> > > > processor and be done with it. Otherwise you'll start programming
> with
> > > > routes, which is something you'd want to avoid.
> > > >
> > > >
> > > > On Fri, Oct 23, 2020, 17:05 Site Register <site.regis...@ymail.com
> > > > .invalid>
> > > > wrote:
> > > >
> > > > > Hi Camel Users,
> > > > > I have a 4G json array file need to load into database. How can I
> > > > leverage
> > > > > Camel to stream the file and split into json?
> > > > > I had tried to use "stream:file" but it was reading line by line
> not
> > > > split
> > > > > into json.
> > > > > I leveraged gson streaming to read and insert into the database in
> java
> > > > > application which took about 3 minutes. However, I would like to
> check if
> > > > > there is a way to leverage camel pipeline for the same purpose.
> > > > > Thank you,
> > > >
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>

Reply via email to