Sebastian T created ARTEMIS-2575:
------------------------------------
Summary: Deadlock in embedded broker during unit test
Key: ARTEMIS-2575
URL: https://issues.apache.org/jira/browse/ARTEMIS-2575
Project: ActiveMQ Artemis
Issue Type: Bug
Components: Broker
Affects Versions: 2.10.1
Reporter: Sebastian T
Today our integration test build of an Spring Boot 2.2 embedded Artemis 2.10.1
failed with a deadlock alert while testing using the CORE protocol. I could not
reproduce the issue again, thus cannot provide a test case but wanted to report
it anyways for review. Maybe based on the stacktraces of the dead locked
threads some conclusion regarding the underlying issue can be drawn.
{code:java}
16:55:49.169 WARN [l-threads)] ionImpl(RemotingConnectionImpl.java:210)
AMQ212037: Connection failure to localhost/127.0.0.1:62616 has been detected:
AMQ219014: Timed out after waiting 30,000 ms for response when sending packet
69 [code=CONNECTION_TIMEDOUT]
16:56:19.171 WARN [l-threads)] ionImpl(RemotingConnectionImpl.java:210)
AMQ212037: Connection failure to localhost/127.0.0.1:62616 has been detected:
AMQ219014: Timed out after waiting 30,000 ms for response when sending packet
69 [code=CONNECTION_TIMEDOUT]
16:56:23.109 WARN [d-threads)] CriticalMeasure(CriticalMeasure.java:99)
Component org.apache.activemq.artemis.core.server.impl.QueueImpl is expired on
path 3
16:56:23.109 WARN [d-threads)] QServerImpl(ActiveMQServerImpl.java:713)
AMQ224081: The component
QueueImpl[name=nonDurable.testQueue.6C9JYR47olzg62aOdBRJIq,
postOffice=PostOfficeImpl
[server=ActiveMQServerImpl::serverUUID=ef2586cd-1dc8-11ea-8bf6-0a5801f021fd],
temp=true]@1ee23908 is not responsive
16:56:23.191 WARN [d-threads)] ServerImpl(ActiveMQServerImpl.java:1022)
AMQ222199: Thread dump:
*******************************************************************************
Complete Thread dump
Deadlock detected!
"Thread-18
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=4061 BLOCKED on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
owned by "Thread-12
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3814
at
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:651)
- blocked on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.sendSessionNotification(ServerSessionImpl.java:455)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:435)
- locked
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@4a68311e
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1597)
at
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:189)
at
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:130)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:1589)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:566)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
Source)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@5014e5b4
"Thread-12
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3814 BLOCKED on
org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908 owned by
"Thread-8
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3768
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.addRedistributor(QueueImpl.java)
- blocked on
org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.onNotification(PostOfficeImpl.java:415)
- locked java.lang.Object@5217b201
at
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.sendNotification(ManagementServiceImpl.java:661)
- locked java.lang.Object@5217b201
- locked
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:600)
- locked
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@7af516ef
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
Source)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@af66fa4
"Thread-8
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3768 BLOCKED on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
owned by "Thread-12
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@3defe5e4)"
Id=3814
at
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl.unregisterQueue(ManagementServiceImpl.java)
- blocked on
org.apache.activemq.artemis.core.server.management.impl.ManagementServiceImpl@2821ce58
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.removeBinding(PostOfficeImpl.java:763)
- locked
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl@7817714f
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2098)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2174)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2122)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2113)
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2093)
at
org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
at
org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$1186/821597250.run(Unknown
Source)
at
org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
at
org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1310)
- locked
org.apache.activemq.artemis.core.server.impl.QueueImpl@1ee23908
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:626)
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:559)
- locked
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@5a799135
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:533)
at
org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1610)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:585)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:285)
at
org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$643/1351406291.onMessage(Unknown
Source)
at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$604/1843989939.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@54c4eef4
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
