James - You’re setting the maxMessagesPerPoll, which will limit the CamelBatchSize to whatever you set that to. So you’re route will shutdown after only processing part of the files in your “batch". If you’re using this to control the number of threads the concurrently work on files, you can set some of the attributes on the <thread> to control that (i.e. poolSize).
You also have sendEmptyMessageWhenIdle set to true - so you could get an empty message. I think you can get what you’re after by removing maxMessagesPerPoll and sendEmptyMessageWhenIdle from the “processProducts” route <from>. As I said, if you also want to limit the number of threads working on files, add the poolSize attribute to the <threads> element. Hopefully that will get what you’re after > On Mar 16, 2016, at 7:06 AM, jamesburn <[email protected]> wrote: > > Hi. Thanks for looking at this. Here are the routes below. > > I think the problem is confounded by my wanting to control processing the > messages asynchronously - we have to cope with a potential 100,000 files. If > I take away <threads> it works as I'd expect, with control-bus stopping the > route, but the messages appear to be processed one at a time. I think it's > perhaps using <threads> which I need to spend time on, but I would like to > know that I can use something like exchangeProperty.CamelBatchIndex to tell > me when they have all finished processing. > > Thanks > James > > <camelContext xmlns="http://camel.apache.org/schema/spring"; > id="ONIXprocess"> > <propertyPlaceholder id="props" > location="file:/opt/apache-servicemix-6.1.0/etc/constant.ONIXprocess.cfg"/> > <route id="headerCollect"> > <from > uri="file:{{onix.header.pickup.location}}?delete=true&include=.*header.*&delay=1000"/> > <log message="starting ProcessProducts route"/> > <to > uri="controlbus:route?routeId=processProducts&action=start"/> > </route> > > <route id="processProducts" autoStartup="false"> > <from > uri="file:{{onix.file.pickup.location}}?delete=true&maxMessagesPerPoll=20&include=.*xml&sendEmptyMessageWhenIdle=true&delay=500"/> > <threads> > > <convertBodyTo type="String"/> > > <choice> > <when> > > <xpath>/Product/ProductIdentifier/IDValue=9780193371644</xpath> > <log message="9780193371644 found in > number > ${exchangeProperty.CamelBatchIndex}"/> > <delay> > <constant>20000</constant> > </delay> > <log message="9780193371644 finished"/> > </when> > </choice> > <to uri="file:{{onix.file.dropoff.location}}"/> > <log message="Processed record index no > ${exchangeProperty.CamelBatchIndex} out of > ${exchangeProperty.CamelBatchSize}"/> > > <choice> > <when> > > <simple>${exchangeProperty.CamelBatchComplete} == 'true'</simple> > <log message="Batch of XML files > finished?: > ${exchangeProperty.CamelBatchComplete}"/> > > > <to > uri="controlbus:route?routeId=processProducts&action=stop"/> > </when> > </choice> > </threads> > </route> > </camelContext> > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/file-trigger-processing-lots-of-files-tp5778807p5779157.html > Sent from the Camel - Users mailing list archive at Nabble.com.
