Vinod Balan created AMQ-7214:
--------------------------------
Summary: ActiveMQ deadlock on upgrading drivers to 5.15.9 from
5.10.0
Key: AMQ-7214
URL: https://issues.apache.org/jira/browse/AMQ-7214
Project: ActiveMQ
Issue Type: Bug
Components: activemq-pool
Affects Versions: 5.15.9
Reporter: Vinod Balan
We tried to upgrade ActiveMQ service from 5.10.10 to 5.15.9.
ActiveMQ service updated *successfully*.
But, The respective jars has also been upgraded in our Tomcat Server
* activemq-client-5.10.0 to activemq-client-5.15.9
* activemq-jms-pool-5.10.0 to activemq-jms-pool-5.15.9
* activemq-pool-5.10.0 to activemq-pool-5.15.9
* hawtbuf-1.10 to hawtbuf-1.11
After the upgrade, While testing with bulk data to analyze performance, we had
an issue where the Messages stopped processing. On analyzing the thread dump,
It was evident that new connections were not being able to create because of a
deadlock.
h2. {color:#cc3300}ActiveMQ Connection Executor:
tcp:///192.168.20.233:61616@57619{color}
priority:5 - threadId:0x000000001e84c000 - nativeId:0xac0 - nativeId
(decimal):2752 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.FifoMessageDispatchChannel.stop({color:#000080}FifoMessageDispatchChannel.java:123{color})
- waiting to lock *<0x0000000087ae1998>* (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageConsumer.stop({color:#000080}ActiveMQMessageConsumer.java:1554{color})
at
org.apache.activemq.ActiveMQSession.stop({color:#000080}ActiveMQSession.java:1863{color})
at
org.apache.activemq.ActiveMQConnection.doStop({color:#000080}ActiveMQConnection.java:586{color})
- locked *<0x0000000084b53290>* (a java.util.concurrent.CopyOnWriteArrayList)
at
org.apache.activemq.ActiveMQConnection.close({color:#000080}ActiveMQConnection.java:642{color})
at
org.apache.activemq.jms.pool.ConnectionPool.close({color:#000080}ConnectionPool.java:176{color})
- locked *<0x0000000084b4a2c8>* (a
org.apache.activemq.pool.PooledConnectionFactory$1)
at
org.apache.activemq.jms.pool.ConnectionPool.onException({color:#000080}ConnectionPool.java:381{color})
at
org.apache.activemq.ActiveMQConnection$5.run({color:#000080}ActiveMQConnection.java:1967{color})
at java.util.concurrent.ThreadPoolExecutor.runWorker({color:#000080}Unknown
Source{color})
at java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#000080}Unknown
Source{color})
at java.lang.Thread.run({color:#000080}Unknown Source{color})
Locked ownable synchronizers:
- *<0x00000000fc2433b8>* (a java.util.concurrent.ThreadPoolExecutor$Worker)
h2. {color:#cc3300}ActiveMQ Session Task-5{color}
priority:7 - threadId:0x0000000031fb7000 - nativeId:0x773c - nativeId
(decimal):30524 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.jms.pool.ConnectionPool.expiredCheck({color:#000080}ConnectionPool.java:223{color})
- waiting to lock *<0x0000000084b4a2c8>* (a
org.apache.activemq.pool.PooledConnectionFactory$1)
at
org.apache.activemq.jms.pool.PooledConnectionFactory$1.validateObject({color:#000080}PooledConnectionFactory.java:129{color})
at
org.apache.activemq.jms.pool.PooledConnectionFactory$1.validateObject({color:#000080}PooledConnectionFactory.java:92{color})
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject({color:#000080}GenericKeyedObjectPool.java:409{color})
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject({color:#000080}GenericKeyedObjectPool.java:284{color})
at
org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection({color:#000080}PooledConnectionFactory.java:235{color})
- locked *<0x0000000084a73be0>* (a
org.apache.activemq.pool.PooledConnectionFactory)
at
org.apache.activemq.jms.pool.PooledConnectionFactory.createConnection({color:#000080}PooledConnectionFactory.java:204{color})
at
com.exterro.fusion.jms.impl.ActiveMQAdapter.publishMessage({color:#000080}ActiveMQAdapter.java:158{color})
at
com.exterro.fusion.jms.impl.ActiveMQAdapter.publishMessage({color:#000080}ActiveMQAdapter.java:132{color})
at
com.exterro.fusion.legalhold.manager.operation.LegalHoldOperationManager.publishBatchMail({color:#000080}LegalHoldOperationManager.java:1082{color})
at
com.exterro.fusion.legalhold.manager.operation.LegalHoldOperationManager.sendLegalHoldCustodianNotification({color:#000080}LegalHoldOperationManager.java:740{color})
at
com.exterro.fusion.legalhold.operation.issue.IssueLegalHoldCustodianEmail.sendNotification({color:#000080}IssueLegalHoldCustodianEmail.java:42{color})
at
com.exterro.fusion.legalhold.operation.common.email.LegalHoldCustodianEmail.execute({color:#000080}LegalHoldCustodianEmail.java:57{color})
at
com.exterro.fusion.legalhold.jms.listeners.common.LegalHoldCustodianEmailListener.executeMessage({color:#000080}LegalHoldCustodianEmailListener.java:70{color})
at
com.exterro.fusion.jms.common.FusionMessagingListener.onMessage({color:#000080}FusionMessagingListener.java:57{color})
at
org.apache.activemq.ActiveMQMessageConsumer.dispatch({color:#000080}ActiveMQMessageConsumer.java:1404{color})
- locked *<0x0000000087ae1998>* (a java.lang.Object)
at
org.apache.activemq.ActiveMQSessionExecutor.dispatch({color:#000080}ActiveMQSessionExecutor.java:131{color})
at
org.apache.activemq.ActiveMQSessionExecutor.iterate({color:#000080}ActiveMQSessionExecutor.java:202{color})
at
org.apache.activemq.thread.PooledTaskRunner.runTask({color:#000080}PooledTaskRunner.java:133{color})
at
org.apache.activemq.thread.PooledTaskRunner$1.run({color:#000080}PooledTaskRunner.java:48{color})
at java.util.concurrent.ThreadPoolExecutor.runWorker({color:#000080}Unknown
Source{color})
at java.util.concurrent.ThreadPoolExecutor$Worker.run({color:#000080}Unknown
Source{color})
at java.lang.Thread.run({color:#000080}Unknown Source{color})
Locked ownable synchronizers:
- *<0x0000000089a89850>* (a java.util.concurrent.ThreadPoolExecutor$Worker)
We use PooledConnectionFactory to create connections and the issue occurs
randomly but with a high frequency.
Please let me know if i can provide any other information for further analysis.
*_Regards,_*
*_Vinod B_*
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
