Hi, The only way I have found to make sure a file or ftp component does not use multiple threads while consuming very large batches is to annotate the QuartzScheduledPollConsumerJob class with @DisallowConcurrentExecution.
That’s it. It seems to work for us. I am not familiar enough with the Camelcode to say what side effects it has and if this would prevent any quartz job in camel to now be single threaded, even if the user does not want it to be. But to me it looks like an oversight when moving from quartz to quartz2. A file or ftp consumer should be single threaded while retrieving. Hope this helps... With kind regards, Hans Orbaan -----Oorspronkelijk bericht----- Van: Hans Orbaan [mailto:[email protected]] Verzonden: Wednesday 23 March 2016 9:52 Aan: [email protected] Onderwerp: RE: Quartz schedule too many workers in route Hi Claus, Did you have a chance to check this already? I really think file and ftp consumers should never use multiple threads unless explicitly defined. This means a from(file://) with a quartz schedule should be stateful by default. Only then would it have the same behaviour as spring scheduler or when using a delay. Thanks! With kind regards, Hans Orbaan -----Oorspronkelijk bericht----- Van: Hans Orbaan [mailto:[email protected]] Verzonden: Monday 21 March 2016 16:52 Aan: [email protected] Onderwerp: RE: Quartz schedule too many workers in route Hi Claus, It was in the email I forwarded. So below my new email. But here it is: from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=#fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler.cron=0+0/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerId=nl_bij-export-to-archive-276) to(file:///data/work/sl/work-archive/work/276/) But the same goes for any component used it looks like (ftp, sftp, etc) Thanks!! With kind regards, Hans Orbaan -----Oorspronkelijk bericht----- Van: Claus Ibsen [mailto:[email protected]] Verzonden: Monday 21 March 2016 16:37 Aan: [email protected] Onderwerp: Re: Quartz schedule too many workers in route On Mon, Mar 21, 2016 at 3:23 PM, Hans Orbaan <[email protected]> wrote: > Hi, > > Is there someone that can help a bit with the issue below? > > An endpoint (file) with a quartz cronschedule is not stateful. This means an > schedule that triggers every minute it will start processing with multiple > threads after 1 minute if it did not finish the job. My quess is this is > unwanted in almost all situations. > > This issue does not happen when spring is used as scheduler. Also this does > not happen when using a regular delay=1m. > > I tried stuff like job.stateful=true, but that is not an allowed option. > At this point I see no other solution then to start using the timer or quartz > component to trigger routes, but I would much rather keep using the file, > ftp, etc components with a schedule. > Ah can you provide an example of the uri you configure for this. We could look into what it would take to make scheduler.job.stateful=true or something being supported. > Any help would be appreciated. > > With kind regards, > > Hans Orbaan > > -----Oorspronkelijk bericht----- > Van: Hans Orbaan [mailto:[email protected]] > Verzonden: Wednesday 16 March 2016 10:18 > Aan: [email protected] > Onderwerp: Quartz schedule too many workers in route > > Hello all, > > We have a few very busy routes that are throwing FileNotFoundExceptions when > processing files (full stacktrace below). Searching for those filenames in > the log it is very clear that those files have already been processed in the > past minute and thus are not there. > After removing the 1 minute schedule and adding a 1 minute delay the problems > have disappeared. > > After testing it looks like a cron scheduler can only start 1 thread/worker > per route and a next scheduler ping will wait for the previous one to be > finished. Even if we manually trigger the route by its triggerId. This was > tested by holding a thread in a debugger for a few minutes, no new threads > passed the breakpoint until we let the first thread go. Still we are seeing > what we are seeing but I cannot create the volume of messages and routes in a > test/develop environment. > > During low volume processing we can see multiple workers busy, but always in > a different minute then the other. So 1 thread/worker is busy per minute but > the worker name can differ. > During high volume processing at some point multiple workers will be > processing, the same route, in the same second. Also there have been moments > we can find 5 camellock files at the same time, in the same directory. There > is only 1 route per directory so there should not be more than 1 camellock, I > think. > > Example from log (all the same routeId): > 2016-03-14 23:35:38,783 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:40,385 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:40,620 [DefaultQuartzScheduler-esbContext-1_Worker-6] > 2016-03-14 23:35:40,843 [DefaultQuartzScheduler-esbContext-1_Worker-8] > 2016-03-14 23:35:41,262 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:41,529 [DefaultQuartzScheduler-esbContext-1_Worker-2] > 2016-03-14 23:35:43,200 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:43,205 [DefaultQuartzScheduler-esbContext-1_Worker-6] > 2016-03-14 23:35:44,470 [DefaultQuartzScheduler-esbContext-1_Worker-8] > 2016-03-14 23:35:44,665 [DefaultQuartzScheduler-esbContext-1_Worker-1] > 2016-03-14 23:35:45,503 [DefaultQuartzScheduler-esbContext-1_Worker-2] > 2016-03-14 23:35:46,393 [DefaultQuartzScheduler-esbContext-1_Worker-7] > 2016-03-14 23:35:47,051 [DefaultQuartzScheduler-esbContext-1_Worker-6] > > The routes are very simple, from a mounted folder to a local directory with > nothing in between except for some onException handling for retrying and > sending to an error route (direct). Lowering maxMessagesPerPoll (250) did > seem to decrease the problem a bit but I am not convinced yet. This should be > tested with a number lower than 100. > > from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter= > #fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler > .cron=0+/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerI > d=nl_bij-export-to-archive-276) > to(file:///data/work/sl/work-archive/work/276/) > > Is there anything we can change to make sure we have only one worker > processing at a time, per route? We seem to have 10 workers total that are > divided amongst the routes with schedulers. We have 250 routes total with a > schedule. Most schedules are every minute. Also there are 250 routes with > delay=5s, but they do not use those workers. > > org.quartz.threadPool.threadCount is now at 1. We might need to increase this > number as we have 250 routes with a 1 minute schedule. But the issue we have > is quite the opposite of what to expect if quartz cannot trigger all jobs it > should trigger. > > ----------------------------------------- > version.camel 2.16.2 > > file:quartz.properties > org.quartz.scheduler.instanceName = QuartzScheduler > org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer = > true org.quartz.scheduler.skipUpdateCheck = true > org.quartz.scheduler.jmx.export = true org.quartz.threadPool.class = > org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount = > 1 org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore > > server.info > Tomcat Version Apache Tomcat/8.0.26 > JVM Version 1.8.0_60-b27 > JVM Vendor Oracle Corporation > OS Name Linux > OS Version 3.8.13-98.2.2.el6uek.x86_64 > OS Architecture amd64 > > 2016-03-14 22:29:02,485 [DefaultQuartzScheduler-esbContext-1_Worker-10] > errors - WARN - Error stacktrace: > org.apache.camel.component.file.GenericFileOperationFailedException: Cannot > store file: /data/work/sl/work-archive/work/276/DD_SOS2016031422110816.xml > at > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292) > at > org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277) > at > org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165) > at > org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79) > at > org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) > at > org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141) > at > org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) > at > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:121) > at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) > at > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) > at > org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442) > at > org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214) > at > org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178) > at > org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174) > at > org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101) > at > org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:59) > at org.quartz.core.JobRunShell.run(JobRunShell.java:202) > at > org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja > va:573) Caused by: org.apache.camel.InvalidPayloadException: No body > available of type: java.io.InputStream but has value: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml] of type: > org.apache.camel.component.file.GenericFile on: DD_SOS2016031422110816.xml. > Caused by: Error during type conversion from type: java.lang.String to the > required type: byte[] with value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory). > Exchange[ID-slesb101-60348-1457682250340-0-642502][DD_SOS2016031422110816.xml]. > Caused by: [org.apache.camel.TypeConversionException - Error during type > conversion from type: java.lang.String to the required type: byte[] with > value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory)] > at > org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101) > at > org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:273) > ... 19 more > Caused by: org.apache.camel.TypeConversionException: Error during type > conversion from type: java.lang.String to the required type: byte[] with > value [Body is file based: > GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]] due > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:177) > at > org.apache.camel.component.file.FileBinding.loadContent(FileBinding.java:57) > at > org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(GenericFileConverter.java:123) > at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > at > org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) > at > org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:99) > ... 20 more > Caused by: org.apache.camel.RuntimeCamelException: > java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at > org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1247) > at > org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59) > at > org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:97) > at sun.reflect.GeneratedMethodAccessor555.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > at > org.apache.camel.impl.converter.StaticMethodFallbackTypeConverter.convertTo(StaticMethodFallbackTypeConverter.java:62) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:333) > at > org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168) > ... 30 more > Caused by: java.io.FileNotFoundException: > /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory) > at java.io.FileInputStream.open0(Native Method) > at java.io.FileInputStream.open(FileInputStream.java:195) > at java.io.FileInputStream.<init>(FileInputStream.java:138) > at > org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:78) > at > org.apache.camel.converter.IOConverter.toByteArray(IOConverter.java:266) > at sun.reflect.GeneratedMethodAccessor604.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243) > ... 39 more -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
