Re: file trigger processing lots of files

[jamesburn]

Hi. Thanks for looking at this. Here are the routes below. 

I think the problem is confounded by my wanting to control processing the
messages asynchronously - we have to cope with a potential 100,000 files. If
I take away <threads> it works as I'd expect, with control-bus stopping the
route, but the messages appear to be processed one at a time. I think it's
perhaps using <threads> which I need to spend time on, but I would like to
know that I can use something like exchangeProperty.CamelBatchIndex to tell
me when they have all finished processing.

Thanks
James

<camelContext xmlns="http://camel.apache.org/schema/spring";
id="ONIXprocess">
        <propertyPlaceholder  id="props"
location="file:/opt/apache-servicemix-6.1.0/etc/constant.ONIXprocess.cfg"/>
        <route id="headerCollect">
                <from
uri="file:{{onix.header.pickup.location}}?delete=true&amp;include=.*header.*&amp;delay=1000"/>
                <log message="starting ProcessProducts route"/>
                <to 
uri="controlbus:route?routeId=processProducts&amp;action=start"/>
        </route>
        
        <route id="processProducts" autoStartup="false">
                <from
uri="file:{{onix.file.pickup.location}}?delete=true&amp;maxMessagesPerPoll=20&amp;include=.*xml&amp;sendEmptyMessageWhenIdle=true&amp;delay=500"/>
                <threads>
                        
                        <convertBodyTo type="String"/>
                        
                        <choice>
                                <when>
                                        
<xpath>/Product/ProductIdentifier/IDValue=9780193371644</xpath> 
                                        <log message="9780193371644 found in 
number
${exchangeProperty.CamelBatchIndex}"/>
                                        <delay>
                                                <constant>20000</constant>
                                        </delay>
                                        <log message="9780193371644 finished"/>
                                </when> 
                        </choice>
                        <to uri="file:{{onix.file.dropoff.location}}"/>
                        <log message="Processed record index no
${exchangeProperty.CamelBatchIndex} out of
${exchangeProperty.CamelBatchSize}"/>   
                        
                        <choice>
                                <when>
                                        
<simple>${exchangeProperty.CamelBatchComplete} == 'true'</simple>
                                        <log message="Batch of XML files 
finished?:
${exchangeProperty.CamelBatchComplete}"/>
                                        
                                                
                                        <to 
uri="controlbus:route?routeId=processProducts&amp;action=stop"/>
                                </when>
                        </choice>
                </threads>
        </route>
</camelContext>



--
View this message in context: 
http://camel.465427.n5.nabble.com/file-trigger-processing-lots-of-files-tp5778807p5779157.html
Sent from the Camel - Users mailing list archive at Nabble.com.