Hello. I have tried different options; like batch size; to evaluate some scenario to optimize some cases. But for cases with a really big volume of data, retrieving them all in memory always leads to an error.
Our current case should be something as simple as : A first route : - receive a soap request from a web client with some kind of filter form to select data - push the xml request to an active queue, and send back a simple soap response A main route : - get back the xml request from the queue - make a json body to set the query from the xml request (10 of 15 lines of groovy for example) - set a header to select the needed collection's attributes - call mongo findAll - marshal the result to csv - write the result into a file. - send a mail to the caller to inform the job is done. This may be done with a very simple blueprint with very few lines and no complexity at all. Do you mean that the only way to process a big volume of Mongo data is to set up a more "smart" algorithm like : - build a first request to count the data. - loop over the data set reading batch parts using "skip" and "page size" - write the paged results appending them to the file. - etc ? Have you an example of paging process ? Thanks for you help. Ephemeris Lappis Le 18/04/2014 02:52, Raul Kripalani [via Camel] a écrit : > Hi, > > We use Mongo cursors to read from the DB. But a DBCursor is not > something we can return to the route because not all technologies > support Streams, Cursors, Chunking, etc. For example, how would you go > about returning a DBCursor to a JMS endpoint? > > That's why we offer the skipping and limiting option so you can > perform pagination in such scenarios. You can also specify a batch > size. Take a look at the component page for further details. > > Hope that helps! > Raúl. > > > On 17 Apr 2014, at 15:41, Ephemeris Lappis <[hidden email] > </user/SendEmail.jtp?type=node&node=5750355&i=0>> wrote: > > > > Hello. > > > > After some tests, it seems that the Camel MongoDB "findAll" > operation tries > > to load all the matching queried data into memory before process > them. With > > collections whose content is about tens millions of documents, this > > naturally leads to OutOfMemoryErrors... > > > > Does this component may use cursors to read the input data and > stream them ? > > > > An idea ? > > > > Thanks in advance. > > > > Regards. > > > > > > > > -- > > View this message in context: > http://camel.465427.n5.nabble.com/Does-Camel-MongoDB-use-cursors-on-findAll-tp5750352.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > ------------------------------------------------------------------------ > If you reply to this email, your message will be added to the > discussion below: > http://camel.465427.n5.nabble.com/Does-Camel-MongoDB-use-cursors-on-findAll-tp5750352p5750355.html > > > To unsubscribe from Does Camel MongoDB use cursors on findAll ?, click > here > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5750352&code=ZXBoZW1lcmlzLmxhcHBpc0BnbWFpbC5jb218NTc1MDM1Mnw0OTQyMjM2NDI=>. > NAML > <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > -- View this message in context: http://camel.465427.n5.nabble.com/Does-Camel-MongoDB-use-cursors-on-findAll-tp5750352p5750357.html Sent from the Camel - Users mailing list archive at Nabble.com.
