Holy moses! A 1 gig JSON doc? Isn't that a criminal offense somewhere? Seriously though, you are hitting a JVM issue more than a camel one. I would see if I could brew up something to shatter the document and have it streamed in and read it piecemeal. You need something like SAX but for JSON. My thinking takes me towards using the Jackson JSON parser in a stream mode. I googled "jackson stream large document" and got some promising hits.
*Robert Simmons Jr. MSc. - Lead Java Architect @ EA* *Author of: Hardcore Java (2003) and Maintainable Java (2012)* *LinkedIn: **http://www.linkedin.com/pub/robert-simmons/40/852/a39 <http://www.linkedin.com/pub/robert-simmons/40/852/a39>* On Sun, Apr 20, 2014 at 8:49 PM, Willem Jiang <willem.ji...@gmail.com>wrote: > Can I have a look at your camel route? > camel-http is using the stream by default and it can spool big streams to > files. > I guess there could be something wrong after you get the response from the > RSET service, So I need to know more detail about your camel route. > > -- > Willem Jiang > > Red Hat, Inc. > Web: http://www.redhat.com > Blog: http://willemjiang.blogspot.com (English) > http://jnn.iteye.com (Chinese) > Twitter: willemjiang > Weibo: 姜宁willem > > > > On April 18, 2014 at 11:02:51 PM, Karthik.Pothuru ( > karthik.poth...@target.com) wrote: > > Dear Team, > > > > We are facing Out of memory issue when receiving a huge json response > (around 1 GB ) when > > hitting a REST webservice with in Target Enterprise. We are currently > using camel Http > > component . > > Due to huge file, the entire data gets loaded into HEAP and the process > is failing due to > > that. > > > > Our requirement is to read this response and store it into a file which > will later be parsed > > for further data processing. > > > > Due to this issue, we are unable to go the next step of parsing. The > same code works fine > > when small volume of data is received through the API. (at present we > remember processing > > a response size of 150 MB and it was successful). > > > > Kindly need your input how to read response with lesser memory foot > print. > > > > Thanks > > Karthik Pothuru > > Mob: +91 91766 97533 > > > >
