I see how #380 is blocked by #308 on lock object 0xfffffffe833683f0, but it looks like #308 is blocked waiting on lock object 0xfffffffe81003250, which isn't held by any of the three threads that are in the partial thread dump you provided. What's the thread dump of whatever thread is the one that's holding that lock?
Also, you may not find much interest in tracking down a deadlock in code that's over 4 years old, because it's possible that the code has changed since 5.9.0 in ways that would fix or avoid this problem. Is upgrading to a current (5.14 or later) version and seeing if the problem reoccurs an option? Tim On Wed, Feb 7, 2018 at 4:01 AM, zyc1984 <yicheng.zh...@murex.com> wrote: > Hi, > > We have identified a potential "deadlock" between BrokerFilter and Producer > creation with VM connection > This block is random, it happened on one given machine but not the others. > > ActiveMQ version 5.9.0 > > Thread #310 : notified by BrokerFilter.addConsumer() and it creates a > Producer with local VM connection > Thread #308 : notifed by BrokerFilter.messageConsumed() and it tries to get > Consumer info in our application with read lock > Thread 380 : it is treating the Producer creation command but it's blocked > by Thread #308 a lock inside held by ActiveMQ BrokerFilter stack. > > > As this issue is not always happening, i guess it is related to the > Threading model in ActiveMQ. > And I would like to know in which case VM connection handling thread will > be > blocked by BrokerFilter and in which case it is not. Is it related to jvm > arguments ? because the configuration of ActiveMQ is hard coded in our > case. > > Thanks, > zyc1984 > > > > > "ActiveMQ Transport: tcp:///22.0.195.149:64315@64078" #310 daemon prio=5 > os_prio=64 tid=0x000000010274e000 nid=0x14d waiting on condition > [0xfffffffe76ffd000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xfffffffe8335fb60> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java > :175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$Condit > ionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403) > at > org.apache.activemq.transport.FutureResponse.getResult(Futur > eResponse.java:40) > at > org.apache.activemq.transport.ResponseCorrelator.request(Res > ponseCorrelator.java:87) > at > org.apache.activemq.ActiveMQConnection.syncSendPacket(Active > MQConnection.java:1394) > at > org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQS > ession.java:1925) > at > org.apache.activemq.ActiveMQMessageProducer.<init>( > ActiveMQMessageProducer.java:125) > at > org.apache.activemq.ActiveMQSession.createProducer(ActiveMQS > ession.java:969) > at EventFeeder.start(GenericEventFeeder.java:63) > at DefaultConsumerManager.startFeed(DefaultConsumerManager. > java:192) > at > DefaultConsumerAdministration.consumerConnected(DefaultConsu > merAdministration.java:249) > at ConsumerListener.onConsumerConnect(ConsumerListener.java:31) > at ActiveMQInterceptor.addConsumer(ActiveMQInterceptor.java:80) > at > org.apache.activemq.broker.MutableBrokerFilter.addConsumer(M > utableBrokerFilter.java:102) > at > org.apache.activemq.broker.TransportConnection.processAddCon > sumer(TransportConnection.java:587) > at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo. > java:347) > at > org.apache.activemq.broker.TransportConnection.service(Trans > portConnection.java:292) > at > org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:149) > at > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > at > org.apache.activemq.transport.WireFormatNegotiator.onCommand > (WireFormatNegotiator.java:113) > at > org.apache.activemq.transport.AbstractInactivityMonitor.onCo > mmand(AbstractInactivityMonitor.java:270) > at > org.apache.activemq.transport.TransportSupport.doConsume(Tra > nsportSupport.java:83) > at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran > sport.java:214) > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > at java.lang.Thread.run(Thread.java:745) > > > > "ActiveMQ Transport: tcp:///22.0.195.149:64313@64078" #308 daemon prio=5 > os_prio=64 tid=0x00000001028d5000 nid=0x14b waiting on condition > [0xfffffffe773fd000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xfffffffe81003250> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java > :175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn > dCheckInterrupt(AbstractQueuedSynchronizer.java:836) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu > ireShared(AbstractQueuedSynchronizer.java:967) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > eShared(AbstractQueuedSynchronizer.java:1283) > at > java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock. > lock(ReentrantReadWriteLock.java:727) > at > DefaultConsumerAdministration.consumed(DefaultConsumerAdmini > stration.java:367) > at ConsumerListener.onMessageConsumed(ConsumerListener.java:45) > at ActiveMQInterceptor.messageConsumed(ActiveMQInterceptor. > java:123) > at > org.apache.activemq.broker.MutableBrokerFilter.messageConsum > ed(MutableBrokerFilter.java:350) > at > org.apache.activemq.broker.region.BaseDestination.messageCon > sumed(BaseDestination.java:465) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue. > java:917) > at org.apache.activemq.broker.region.Queue.removeMessage(Queue. > java:1703) > at > org.apache.activemq.broker.region.QueueSubscription.acknowle > dge(QueueSubscription.java:63) > at > org.apache.activemq.broker.region.PrefetchSubscription.ackno > wledge(PrefetchSubscription.java:235) > - locked <0xfffffffe833683f0> (a java.lang.Object) > at > org.apache.activemq.broker.region.AbstractRegion.acknowledge > (AbstractRegion.java:412) > at > org.apache.activemq.broker.region.RegionBroker.acknowledge( > RegionBroker.java:458) > at > org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82) > at > org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82) > at > org.apache.activemq.broker.TransactionBroker.acknowledge(Tra > nsactionBroker.java:277) > at > org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82) > at > org.apache.activemq.broker.MutableBrokerFilter.acknowledge(M > utableBrokerFilter.java:92) > at > org.apache.activemq.broker.TransportConnection.processMessag > eAck(TransportConnection.java:476) > at org.apache.activemq.command.MessageAck.visit(MessageAck.java > :236) > at > org.apache.activemq.broker.TransportConnection.service(Trans > portConnection.java:292) > at > org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:149) > at > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > at > org.apache.activemq.transport.WireFormatNegotiator.onCommand > (WireFormatNegotiator.java:113) > at > org.apache.activemq.transport.AbstractInactivityMonitor.onCo > mmand(AbstractInactivityMonitor.java:270) > at > org.apache.activemq.transport.TransportSupport.doConsume(Tra > nsportSupport.java:83) > at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran > sport.java:214) > at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - None > > > > "ActiveMQ VMTransport: vm://BOS_JMS_BROKER#1-3" #380 daemon prio=5 > os_prio=64 tid=0x0000000102bad000 nid=0x18a waiting for monitor entry > [0xfffffffe763fd000] > java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.activemq.broker.region.PrefetchSubscription.dispa > tchPending(PrefetchSubscription.java:654) > - waiting to lock <0xfffffffe833683f0> (a java.lang.Object) > - locked <0xfffffffe83368400> (a java.lang.Object) > at > org.apache.activemq.broker.region.PrefetchSubscription.add( > PrefetchSubscription.java:161) > at > org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1998) > at org.apache.activemq.broker.region.Queue.doDispatch(Queue.jav > a:1946) > at org.apache.activemq.broker.region.Queue.pageInMessages(Queue > .java:2086) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java: > 1581) > - locked <0xfffffffe83368658> (a java.lang.Object) > at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803) > at org.apache.activemq.broker.region.Queue.messageSent(Queue. > java:1797) > at org.apache.activemq.broker.region.Queue.doMessageSend(Queue. > java:866) > at org.apache.activemq.broker.region.Queue.send(Queue.java:700) > at > org.apache.activemq.broker.region.AbstractRegion.send(Abstra > ctRegion.java:394) > at > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:442) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter. > java:147) > at > org.apache.activemq.broker.CompositeDestinationBroker.send(C > ompositeDestinationBroker.java:96) > at > org.apache.activemq.broker.TransactionBroker.send(Transactio > nBroker.java:307) > at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter. > java:147) > at > org.apache.activemq.broker.MutableBrokerFilter.send(MutableB > rokerFilter.java:152) > at > org.apache.activemq.broker.TransportConnection.processMessag > e(TransportConnection.java:467) > at > org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMe > ssage.java:751) > at > org.apache.activemq.broker.TransportConnection.service(Trans > portConnection.java:292) > at > org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:149) > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(R > esponseCorrelator.java:116) > at > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > at > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247) > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTa > skRunner.java:129) > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTask > Runner.java:47) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > Locked ownable synchronizers: > - <0xfffffffe833687b8> (a java.util.concurrent.ThreadPoo > lExecutor$Worker) > - <0xfffffffe83407310> (a > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) > > > > > > -- > Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805 > .html >
