Lee Kong created CAMEL-9994: ------------------------------- Summary: Deadlock in Failover Loadbalancer Key: CAMEL-9994 URL: https://issues.apache.org/jira/browse/CAMEL-9994 Project: Camel Issue Type: Bug Components: camel-core Affects Versions: 2.17.1, 2.16.1, 2.15.3 Environment: Mac OSX + ActiveMQ, Solaris + MQ Reporter: Lee Kong
Calling the route below with 2 threads simultaneously, when the messages timeout, I get a deadlock. The issue seems to be with CorrelationTimeoutMap.onEviction() where a thread is trying to acquire a lock while holding another lock (another thread tries to do the same). I have created a simple project to demonstrate the issue: https://github.com/leechinkong/CamelFailoverLoadbalancer from("direct:test") .loadBalance() .failover(3, true, true) .inOut("jms:testRqQueue1?replyTo=testRsQueue1&useMessageIDAsCorrelationID=true","jms:testRqQueue2?replyTo=testRsQueue2&useMessageIDAsCorrelationID=true"); Thread dump: "Camel (camel-1) thread #0 - JmsReplyManagerTimeoutChecker[testRsQueue1]": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007aca28f60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) at org.apache.camel.support.DefaultTimeoutMap.putIfAbsent(DefaultTimeoutMap.java:116) at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.putIfAbsent(CorrelationTimeoutMap.java:108) at org.apache.camel.component.jms.reply.ReplyManagerSupport.registerReply(ReplyManagerSupport.java:105) at org.apache.camel.component.jms.JmsProducer$1.createMessage(JmsProducer.java:227) at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:282) at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:225) at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$1.doInJms(JmsConfiguration.java:239) at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:493) at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:236) at org.apache.camel.component.jms.JmsProducer.doSend(JmsProducer.java:409) at org.apache.camel.component.jms.JmsProducer.processInOut(JmsProducer.java:239) at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:150) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91) at org.apache.camel.builder.NoErrorHandlerBuilder$1.process(NoErrorHandlerBuilder.java:40) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) at org.apache.camel.processor.Pipeline.process(Pipeline.java:118) at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51) at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190) at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:49) at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:896) at org.apache.camel.processor.RedeliveryErrorHandler.processAsyncErrorHandler(RedeliveryErrorHandler.java:561) at org.apache.camel.processor.RedeliveryErrorHandler$1.done(RedeliveryErrorHandler.java:466) at org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86) at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:139) at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250) at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:145) at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:250) at org.apache.camel.builder.NoErrorHandlerBuilder$1$1.done(NoErrorHandlerBuilder.java:44) at org.apache.camel.management.InstrumentationProcessor$1.done(InstrumentationProcessor.java:86) at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:139) at org.apache.camel.component.jms.reply.ReplyManagerSupport.processReply(ReplyManagerSupport.java:179) at org.apache.camel.component.jms.reply.TemporaryQueueReplyHandler.onTimeout(TemporaryQueueReplyHandler.java:61) at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:54) at org.apache.camel.component.jms.reply.CorrelationTimeoutMap.onEviction(CorrelationTimeoutMap.java:30) at org.apache.camel.support.DefaultTimeoutMap.purge(DefaultTimeoutMap.java:230) at org.apache.camel.support.DefaultTimeoutMap.run(DefaultTimeoutMap.java:180) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)