[
https://issues.apache.org/jira/browse/ARTEMIS-2575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997467#comment-16997467
]
Sebastian T commented on ARTEMIS-2575:
--------------------------------------
Based on the stacktrace it looks like a race condition between two consumers
being closed at the exact same time on the same queue.
Thread 12 makes it to ServerConsumerImpl line 600 and Thread 8 makes it to
ServerConsumerImpl line 559. Then the deadlock happens because Thread 12 has a
lock on ManagementServcieImpl and tries to acquire a lock on the queue object.
Whereas Thread 8 has a lock on the queue object and tries to acquire a lock on
ManagementServcieImpl.
"Thread-12
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3814 BLOCKED on
org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908
owned by "Thread-8
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3768
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.addRedistributor(QueueImpl.java)
- *blocked {color:#00875a}on
org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908{color}*
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:415)
- locked java.lang.Object@5217b201
at
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:661)
- locked java.lang.Object@5217b201
- *locked{color:#ff8b00}
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58{color}*
*at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:600)*
** - locked
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@7af516ef
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
Source)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
...
"Thread-8
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3768 BLOCKED on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
owned by "Thread-12
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3814
at
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.unregisterQueue(ManagementServiceImpl.java)
- *blocked{color:#ff8b00} on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58{color}*
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:763)
- locked
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl@7817714f
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2098)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2174)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2122)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2113)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2093)
at
org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
at
org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$1186/821597250.run(Unknown
Source)
at
org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
at
org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1310)
- *locked
{color:#00875a}org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908{color}*
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:626)
*at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:559)*
** - locked
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@5a799135
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
Source)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
...
> Deadlock in embedded broker during unit test
> --------------------------------------------
>
> Key: ARTEMIS-2575
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2575
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.10.1
> Reporter: Sebastian T
> Priority: Minor
>
> Today our integration test build of an Spring Boot 2.2 embedded Artemis
> 2.10.1 failed with a deadlock alert while testing using the CORE protocol. I
> could not reproduce the issue again, thus cannot provide a test case but
> wanted to report it anyways for review. Maybe based on the stacktraces of the
> dead locked threads some conclusion regarding the underlying issue can be
> drawn.
>
> {code:java}
> 16:55:49.169 WARN [l-threads)] ionImpl(RemotingConnectionImpl.java:210)
> AMQ212037: Connection failure to localhost/127.0.0.1:62616 has been detected:
> AMQ219014: Timed out after waiting 30,000 ms for response when sending packet
> 69 [code=CONNECTION_TIMEDOUT]
> 16:56:19.171 WARN [l-threads)] ionImpl(RemotingConnectionImpl.java:210)
> AMQ212037: Connection failure to localhost/127.0.0.1:62616 has been detected:
> AMQ219014: Timed out after waiting 30,000 ms for response when sending packet
> 69 [code=CONNECTION_TIMEDOUT]
> 16:56:23.109 WARN [d-threads)] CriticalMeasure(CriticalMeasure.java:99)
> Component org.apache.activemq.artemis.core.server.impl.QueueImpl is expired
> on path 3
> 16:56:23.109 WARN [d-threads)] QServerImpl(ActiveMQServerImpl.java:713)
> AMQ224081: The component
> QueueImpl[name=nonDurable.testQueue.6C9JYR47olzg62aOdBRJIq,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=ef2586cd-1dc8-11ea-8bf6-0a5801f021fd],
> temp=true]@1ee23908 is not responsive
> 16:56:23.191 WARN [d-threads)] ServerImpl(ActiveMQServerImpl.java:1022)
> AMQ222199: Thread dump:
> *******************************************************************************
> Complete Thread dump
> Deadlock detected!
> "Thread-18
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=4061 BLOCKED on
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
> owned by "Thread-12
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=3814
> at
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:651)
> - blocked on
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.sendSessionNotification(ServerSessionImpl.java:455)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:435)
> - locked
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@4a68311e
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1597)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:189)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:130)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:1589)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:566)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
> Source)
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@5014e5b4
> "Thread-12
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=3814 BLOCKED on
> org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908 owned by
> "Thread-8
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=3768
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.addRedistributor(QueueImpl.java)
> - blocked on
> org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:415)
> - locked java.lang.Object@5217b201
> at
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:661)
> - locked java.lang.Object@5217b201
> - locked
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:600)
> - locked
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@7af516ef
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
> Source)
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@af66fa4
> "Thread-8
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=3768 BLOCKED on
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
> owned by "Thread-12
> (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
> Id=3814
> at
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.unregisterQueue(ManagementServiceImpl.java)
> - blocked on
> org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:763)
> - locked
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl@7817714f
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2098)
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2174)
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2122)
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2113)
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2093)
> at
> org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
> at
> org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$1186/821597250.run(Unknown
> Source)
> at
> org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
> at
> org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1310)
> - locked
> org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:626)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:559)
> - locked
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@5a799135
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
> at
> org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
> Source)
> at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@54c4eef4
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
