Hi Maybe poll from the database with a limited set of data per poll.
Also is 1 row in the database = 1 xml Then you can just poll one row at a time, not as efficient but it gets the job done. On Fri, Apr 11, 2014 at 9:58 AM, davidrobert <[email protected]> wrote: > We have the following requirement: > 1. Poll from a database > 2. Marshall the result set into XML > 3. Send the XML in a JMS message to a downstream system > > The issue is, some of the result sets could be huge - millions of rows. The > transformer may be unable to handle this simply because of memory/heap space > limitations. There is also the issue of sending a massive message. > > Does Camel have an easy solution for this, some way of streaming the result > set to a transformer in small chunks, then sending individual messages for > each piece which are identified as part of a collection by some kind of > correlation id or something similar? > > Or is our requirement simply not suited for messaging infrastructure, in > which case we should look at a solution based around batch processing? > > Thanks > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096.html > Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen ----------------- Red Hat, Inc. Email: [email protected] Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen hawtio: http://hawt.io/ fabric8: http://fabric8.io/
