[
https://issues.apache.org/jira/browse/AMQ-8416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Geoff Denning updated AMQ-8416:
-------------------------------
Description:
Using the ActiveMQ Failover Transport, we're seeing issues where our
application randomly stops receiving messages from ActiveMQ until the
application is restarted.
We ran a thread dump, and saw that a Java-level deadlock was detected between
two ActiveMQ threads:
{code:java}
Found one Java-level deadlock:
=============================
"ActiveMQ Task-1":
waiting to lock monitor 0x00007f8286d91a18 (object 0x00000000c06966d8, a
java.lang.Object),
which is held by "ActiveMQ Task-3"
"ActiveMQ Task-3":
waiting to lock monitor 0x00007f8286d91ac8 (object 0x00000000c0afb960, a
java.lang.Object),
which is held by "ActiveMQ Task-1"Java stack information for the threads
listed above:
===================================================
"ActiveMQ Task-1":
at
org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:274)
- waiting to lock <0x00000000c06966d8> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$3.onException(FailoverTransport.java:224)
at
org.apache.activemq.transport.failover.FailoverTransport.propagateFailureToExceptionListener(FailoverTransport.java:1147)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1096)
- locked <0x00000000c0afb960> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
- locked <0x00000000c0afb998> (a java.lang.Object)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"ActiveMQ Task-3":
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x00000000c0afb960> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:853)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1022)
- locked <0x00000000c06966d8> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
- locked <0x00000000c06966e8> (a java.lang.Object)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock. {code}
was:
Using the ActiveMQ Failover Transport, we're seeing issues where our
application stops receiving messages from ActiveMQ. We ran a thread dump, and
saw that a Java-level deadlock was detected between two ActiveMQ threads:
{code:java}
Found one Java-level deadlock:
=============================
"ActiveMQ Task-1":
waiting to lock monitor 0x00007f8286d91a18 (object 0x00000000c06966d8, a
java.lang.Object),
which is held by "ActiveMQ Task-3"
"ActiveMQ Task-3":
waiting to lock monitor 0x00007f8286d91ac8 (object 0x00000000c0afb960, a
java.lang.Object),
which is held by "ActiveMQ Task-1"Java stack information for the threads
listed above:
===================================================
"ActiveMQ Task-1":
at
org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:274)
- waiting to lock <0x00000000c06966d8> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$3.onException(FailoverTransport.java:224)
at
org.apache.activemq.transport.failover.FailoverTransport.propagateFailureToExceptionListener(FailoverTransport.java:1147)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1096)
- locked <0x00000000c0afb960> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
- locked <0x00000000c0afb998> (a java.lang.Object)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"ActiveMQ Task-3":
at java.lang.Object.wait(Native Method)
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
- locked <0x00000000c0afb960> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:853)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1022)
- locked <0x00000000c06966d8> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
- locked <0x00000000c06966e8> (a java.lang.Object)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock. {code}
> Deadlock during failover transport reconnect
> --------------------------------------------
>
> Key: AMQ-8416
> URL: https://issues.apache.org/jira/browse/AMQ-8416
> Project: ActiveMQ
> Issue Type: Bug
> Components: Transport
> Affects Versions: 5.16.0
> Environment: Amazon Linux 2, Apache Tomcat, Java 8
> Reporter: Geoff Denning
> Priority: Major
>
> Using the ActiveMQ Failover Transport, we're seeing issues where our
> application randomly stops receiving messages from ActiveMQ until the
> application is restarted.
> We ran a thread dump, and saw that a Java-level deadlock was detected between
> two ActiveMQ threads:
> {code:java}
> Found one Java-level deadlock:
> =============================
> "ActiveMQ Task-1":
> waiting to lock monitor 0x00007f8286d91a18 (object 0x00000000c06966d8, a
> java.lang.Object),
> which is held by "ActiveMQ Task-3"
> "ActiveMQ Task-3":
> waiting to lock monitor 0x00007f8286d91ac8 (object 0x00000000c0afb960, a
> java.lang.Object),
> which is held by "ActiveMQ Task-1"Java stack information for the threads
> listed above:
> ===================================================
> "ActiveMQ Task-1":
> at
> org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:274)
> - waiting to lock <0x00000000c06966d8> (a java.lang.Object)
> at
> org.apache.activemq.transport.failover.FailoverTransport$3.onException(FailoverTransport.java:224)
> at
> org.apache.activemq.transport.failover.FailoverTransport.propagateFailureToExceptionListener(FailoverTransport.java:1147)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1096)
> - locked <0x00000000c0afb960> (a java.lang.Object)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
> - locked <0x00000000c0afb998> (a java.lang.Object)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "ActiveMQ Task-3":
> at java.lang.Object.wait(Native Method)
> at
> org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:623)
> - locked <0x00000000c0afb960> (a java.lang.Object)
> at
> org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:853)
> at
> org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1022)
> - locked <0x00000000c06966d8> (a java.lang.Object)
> at
> org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
> - locked <0x00000000c06966e8> (a java.lang.Object)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Found 1 deadlock. {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
