Re: Camel quartz misfires and route not getting run / triggered after exception:ObjectAlreadyExistsException
Hi, Can you kindly explain why the route starts failing after running for sometime.. Why is the doAddJob() getting called, after few runs of a schedule have already tun, as reported in the exception? The race condition can happen only at the start of the route, while scheduling the quartz job. But it gets reported after few runs in the below exception trace: http://camel.465427.n5.nabble.com/file/n5743232/Route_Stoping_After_fewruns.png Thanks, Lakshmi Exception Trace: ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null# Error occurred during starting Camel: CamelContext(context4) due Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification.org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification. at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200) at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788) at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712) at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059) at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822) at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) at org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) at org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) at org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) at org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) -- View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5743232.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered after exception:ObjectAlreadyExistsException
Hi As you've already reported by your first post, only increasing the number of running VM's makes this exception to happen (from 2 to 7). This CAN be a hint for a race-condition somewhere. There was a similar problem being reported while ago: https://issues.apache.org/jira/browse/CAMEL-6686 The only reason why we could fix that was because we could reproduce the same exception inside a test, which is now added to the code base of this Camel module. In your case I was NOT able to do the same. If you could somehow provide a test or a sample project then we could dig into this to see if we can find the root cause of the problem and probably fix it. There's also the following Camel test module you could make use of for this: http://camel.apache.org/blueprint-testing.html Babak lakshmi.prashant wrote Hi, Can you kindly explain why the route starts failing after running for sometime.. Why is the doAddJob() getting called, after few runs of a schedule have already tun, as reported in the exception? The race condition can happen only at the start of the route, while scheduling the quartz job. But it gets reported after few runs in the below exception trace: http://camel.465427.n5.nabble.com/file/n5743232/Route_Stoping_After_fewruns.png Thanks, Lakshmi Exception Trace: ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null# Error occurred during starting Camel: CamelContext(context4) due Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification.org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification. at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200) at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788) at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712) at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059) at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822) at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) at org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) at org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) at org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) at org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) -- View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5743238.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel quartz misfires and route not getting run / triggered after exception:ObjectAlreadyExistsException
Hi
Thanks for getting back. Your Quartz configuration seems fine.
To me this sounds like a bug in Quartz itself as Quartz claims that the
trigger is null/non-existing but when we try to schedule the Job using that
given trigger it blows up with ObjectAlreadyExistsException.
That said there's a race condition between the time window as we ask for a
pre-existing trigger and the point where we try to schedule the Job. What if
in this time window another JVM (a cluster member) schedules a Job using the
same trigger name/group. IMHO what actually would come handy here would be a
*atomic* Quartz-API for this, something like what we already have in Java
itself:
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ConcurrentHashMap.html#putIfAbsent(K,%20V)
Which as it's Javadoc already says is equivalent to:
if (!map.containsKey(key))
return map.put(key, value);
else
return map.get(key);
Except that the action is performed *atomically*.
Maybe you want to ask for help using the Quartz user-forum itself and refer
to this thread?
Babak
lakshmi.prashant wrote
Hi,
Sorry for the delay in response. I did not get notified in my mail box
and hence had not followed your reply to the mail thread.
Please find below my answers and the blueprint xml used in the cluster
with 7 VM's.
Thanks,
Lakshmi
- Which exact Quartz 1.x version do you make use of? :
/
Ans: I am using servicemix quartz 1.8.6_1 bundle.
/
- Can you reproduce this with the latest Camel 2.12.1 version?
/
Ans: Yes. It happens in big clusters. Has any fix been provided with camel
2.12.1?
/
- And CAMEL-5994 is already fixed so this should not cause any problem for
you
- Also looking at the stacktrace I think Quartz has not been setup
properly in Cluster-Mode (see
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
on the call-stack)... Did you set the org.quartz.jobStore.isClustered and
org.quartz.jobStore.clusterCheckinInterval properties appropriately?
/
Ans: Please find the blueprint XML that I had used. I have set isClustered
as true and the logs showed that the route got triggered by quartz in
different cluster VM's for different runs, till the exception got
reported. But I was not sure of the right value for the checkin interval
in the cluster and had not specified any specific value.
/
- As you run inside OSGi but assign the id of
camelContext
by yourself (e.g. context4), have you seen the blue-box about this here
http://camel.apache.org/quartz
/
Ans: I am assigning an unique id for each bundle with the camel route. As
of now, I had tested with only 1 bundle deployed in multiple cluster nodes
hence I guess that the context id should not be a problem.
I will also check by removing the context id, as well.
/
I had a look at the quartz component source code: The error trace shows
that existingTrigger is null. It looks like that a race condition is
occuring between the different VM's. Both of them are likely getting the
trigger name as null and try to schedule the job..but the job is already
scheduled
private void doAddJob(JobDetail job, Trigger trigger) throws
SchedulerException {
incrementJobCounter(getScheduler());
Trigger existingTrigger =
getScheduler().getTrigger(trigger.getName(), trigger.getGroup());
if (existingTrigger == null) {
LOG.debug(Adding job using trigger: {}/{},
trigger.getGroup(), trigger.getName());
*
getScheduler().scheduleJob(job, trigger);
*
} else if (hasTriggerChanged(existingTrigger, trigger)) {
LOG.debug(Trigger: {}/{} already exists and will be updated
by Quartz., trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any
misfire to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name, which
is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(), trigger.getGroup(),
trigger);
} else {
if (!isClustered()) {
LOG.debug(Trigger: {}/{} already exists and will be
resumed by Quartz., trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any
misfire to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name,
which is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(),
trigger.getGroup(), trigger);
} else {
LOG.debug(Trigger: {}/{} already exists and is already
scheduled by clustered JobStore., trigger.getGroup(),
Re: Camel quartz misfires and route not getting run / triggered after exception:ObjectAlreadyExistsException
Hi,
Sorry for the delay in response. I did not get notified in my mail box and
hence had not followed your reply to the mail thread.
Please find below my answers and the blueprint xml used in the cluster with
7 VM's.
Thanks,
Lakshmi
- Which exact Quartz 1.x version do you make use of? :
/Ans: I am using servicemix quartz 1.8.6_1 bundle./
- Can you reproduce this with the latest Camel 2.12.1 version?
/Ans: Yes. It happens in big clusters. Has any fix been provided with camel
2.12.1?/
- And CAMEL-5994 is already fixed so this should not cause any problem for
you
- Also looking at the stacktrace I think Quartz has not been setup properly
in Cluster-Mode (see
org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232)
on the call-stack)... Did you set the org.quartz.jobStore.isClustered and
org.quartz.jobStore.clusterCheckinInterval properties appropriately?
/Ans: Please find the blueprint XML that I had used. I have set isClustered
as true and the logs showed that the route got triggered by quartz in
different cluster VM's for different runs, till the exception got reported.
But I was not sure of the right value for the checkin interval in the
cluster and had not specified any specific value./
- As you run inside OSGi but assign the id of camelContext by yourself
(e.g. context4), have you seen the blue-box about this here
http://camel.apache.org/quartz
/Ans: I am assigning an unique id for each bundle with the camel route. As
of now, I had tested with only 1 bundle deployed in multiple cluster nodes
hence I guess that the context id should not be a problem.
I will also check by removing the context id, as well.
/
I had a look at the quartz component source code: The error trace shows
that existingTrigger is null. It looks like that a race condition is
occuring between the different VM's. Both of them are likely getting the
trigger name as null and try to schedule the job..but the job is already
scheduled
private void doAddJob(JobDetail job, Trigger trigger) throws
SchedulerException {
incrementJobCounter(getScheduler());
Trigger existingTrigger =
getScheduler().getTrigger(trigger.getName(), trigger.getGroup());
if (existingTrigger == null) {
LOG.debug(Adding job using trigger: {}/{}, trigger.getGroup(),
trigger.getName());
*getScheduler().scheduleJob(job, trigger);*
} else if (hasTriggerChanged(existingTrigger, trigger)) {
LOG.debug(Trigger: {}/{} already exists and will be updated by
Quartz., trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any misfire
to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name, which
is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(), trigger.getGroup(),
trigger);
} else {
if (!isClustered()) {
LOG.debug(Trigger: {}/{} already exists and will be resumed
by Quartz., trigger.getGroup(), trigger.getName());
// fast forward start time to now, as we do not want any
misfire to kick in
trigger.setStartTime(new Date());
// replace job, and relate trigger to previous job name,
which is needed to reschedule job
scheduler.addJob(job, true);
trigger.setJobName(existingTrigger.getJobName());
scheduler.rescheduleJob(trigger.getName(),
trigger.getGroup(), trigger);
} else {
LOG.debug(Trigger: {}/{} already exists and is already
scheduled by clustered JobStore., trigger.getGroup(), trigger.getName());
}
}
}
*Blueprint XML used by me:*
?xml version=1.0 encoding=UTF-8?
blueprint xmlns=http://www.osgi.org/xmlns/blueprint/v1.0.0;
xmlns:Generator=com.sap.it.gnb.ifl.common.gen.pluggability.api.GeneratorHandler
xmlns:str=http://exslt.org/strings;
xmlns:exsl=http://exslt.org/common;
xmlns:ext=http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0;
xmlns:idocsoap=urn:sap-com:document:sap:idoc:soap:messages
xmlns:bsn=http://sapcd.com/bsnagt;
xmlns:ns=https://bsnschemas.netweaver.neo.com/bsnflow;
xmlns:jaxws=http://cxf.apache.org/blueprint/jaxws;
xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance;
xmlns:sec=http://cxf.apache.org/configuration/security;
xmlns:wsrm-policy=http://schemas.xmlsoap.org/ws/2005/02/rm/policy;
xmlns:wsrm-mgr=http://cxf.apache.org/ws/rm/manager;
xmlns:wsa=http://cxf.apache.org/ws/addressing;
xmlns:http-conf=http://cxf.apache.org/transports/http/configuration;
xmlns:cxfcore=http://cxf.apache.org/blueprint/core;
Re: Camel quartz misfires and route not getting run / triggered after exception:ObjectAlreadyExistsException
Hi Were you able to find the root cause of the problem here? If not: - Which exact Quartz 1.x version do you make use of? - Can you reproduce this with the latest Camel 2.12.1 version? - And CAMEL-5994 is already fixed so this should not cause any problem for you - Also looking at the stacktrace I think Quartz has not been setup properly in Cluster-Mode (see org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) on the call-stack)... Did you set the org.quartz.jobStore.isClustered and org.quartz.jobStore.clusterCheckinInterval properties appropriately? - As you run inside OSGi but assign the id of camelContext by yourself (e.g. context4), have you seen the blue-box about this here http://camel.apache.org/quartz Babak lakshmi.prashant wrote Hi, I am running camel quartz (2.10.4) quartz has been set-up in clustered mode. The clocks in the cluster are synchronized. I have set-up a trigger to run every 1 minute, via the camel-quartz end-point in my route. a) it works fine if quartz is not set up in clustered mode (uses RAMJobstore). b) If quartz is set up in clustred mode, it was working fine when there were only 2 nodes in the cluster. Sometimes, instead of every 1 minute, the runs used to get fired every 2 minutes over a time-window, but it seemed to stabilize and run fine. c) After I increased the number of VM's in the cluster (to ~7 VM's), I faced misfires.. i) The routes used to be triggered correctly for sometime and then stop working altogether.. ii) The below exception has been logged in 1 (or) more VM's in the cluster, after which the camel endpoints seemed to not fire at all. PFA the depiction of misfires from camel. Note: This issue is noticed in 2.11 camel as well. Is this related to https://issues.apache.org/jira/browse/CAMEL-5994... And has this not been fixed till now? Kindly help. Thanks, Lakshmi Exception Trace: ERROR#org.apache.camel.blueprint.BlueprintCamelContext##anonymous#Blueprint Extender: 3##avatarcl#aq4appaq4t#iflmap#null#null#Error occurred during starting Camel: CamelContext(context4) due Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification.org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'schedule3' and group: 'Camel', because one already exists with this identification. at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeTrigger(JobStoreSupport.java:1200) at org.quartz.impl.jdbcjobstore.JobStoreSupport$3.execute(JobStoreSupport.java:1072) at org.quartz.impl.jdbcjobstore.JobStoreSupport$40.execute(JobStoreSupport.java:3716) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3788) at org.quartz.impl.jdbcjobstore.JobStoreTX.executeInLock(JobStoreTX.java:90) at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInLock(JobStoreSupport.java:3712) at org.quartz.impl.jdbcjobstore.JobStoreSupport.storeJobAndTrigger(JobStoreSupport.java:1059) at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:822) at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243) at org.apache.camel.component.quartz.QuartzComponent.doAddJob(QuartzComponent.java:232) at org.apache.camel.component.quartz.QuartzComponent.addJob(QuartzComponent.java:222) at org.apache.camel.component.quartz.QuartzEndpoint.addTrigger(QuartzEndpoint.java:81) at org.apache.camel.component.quartz.QuartzEndpoint.consumerStarted(QuartzEndpoint.java:213) at org.apache.camel.component.quartz.QuartzConsumer.doStart(QuartzConsumer.java:39) at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:1819) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:2113) at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers(DefaultCamelContext.java:2049) at org.apache.camel.impl.DefaultCamelContext.safelyStartRouteServices(DefaultCamelContext.java:1979) at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRoutes(DefaultCamelContext.java:1758) at org.apache.camel.impl.DefaultCamelContext.doStartCamel(DefaultCamelContext.java:1633) at org.apache.camel.impl.DefaultCamelContext.doStart(DefaultCamelContext.java:1500) http://camel.465427.n5.nabble.com/file/n5739997/FailedRuns.jpg http://camel.465427.n5.nabble.com/file/n5739997/runs_successful_with_2_VMs.jpg http://camel.465427.n5.nabble.com/file/n5739997/some_runs_misfiring_fire_in_2_mins.jpg -- View this message in context: http://camel.465427.n5.nabble.com/Camel-quartz-misfires-and-route-not-getting-run-triggered-after-exception-ObjectAlreadyExistsExceptin-tp5739997p5740345.html Sent
