Deadlock with static:(failover:(...)) configuration
---------------------------------------------------
Key: AMQ-3277
URL: https://issues.apache.org/jira/browse/AMQ-3277
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.2
Environment: 5.4.2 with back-ported patches for AMQ-3176, AMQ-2736,
AMQ-3124, AMQ-3125, AMQ-3167.
Two brokers, each with a single duplex network connection to the other, using
static:(failover:(first,second)) URI format.
Reporter: Arthur Naseef
Using failover transports with static transports to create a network of brokers
with H/A pairs, the failovers lead to deadlocks.
This appears related to AMQ-3176, but the patch for that problem is included in
our 5.4.2 build.
<networkConnector name="nw1_from_nw2"
uri="static:(failover:(tcp://127.0.0.1:60992,tcp://127.0.0.1:60002))"
duplex="true" conduitSubscriptions="false"
dynamicOnly="true"
networkTTL="7"
suppressDuplicateQueueSubscriptions="true"
decreaseNetworkConsumerPriority="true"
bridgeTempDestinations="true"
>
</networkConnector>
The deadlock occurs after stopping and restarting each broker once. Here's the
steps:
- start broker1
- start broker2
* test connectivity
- sthudown broker1
- restart broker1
* test connectivity
- shutdown broker2
- restart broker2
* test connectivity
Here is the deadlock information from the JVM:
Java stack information for the threads listed above:
===================================================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
at
org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:477)
- waiting to lock <0x00002aaac2df83c8> (a java.lang.Object)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
- locked <0x00002aaac2e050f8> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.startRemoteBridge(DemandForwardingBridgeSupport.java:352)
- locked <0x00002aaac2df7c50> (a
org.apache.activemq.network.DurableConduitBridge)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$4.run(DemandForwardingBridgeSupport.java:277)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"ActiveMQ Failover Worker: 169776139":
at
org.apache.activemq.network.DemandForwardingBridgeSupport.startLocalBridge(DemandForwardingBridgeSupport.java:290)
- waiting to lock <0x00002aaac2df7c50> (a
org.apache.activemq.networFound one Java-level deadlock:
=============================
"StartRemotelBridge: localBroker=vm://peer-nw2#2":
waiting to lock monitor 0x0000000044796938 (object 0x00002aaac2df83c8, a
java.lang.Object),
which is held by "ActiveMQ Failover Worker: 169776139"
"ActiveMQ Failover Worker: 169776139":
waiting to lock monitor 0x00002aaac470f0b0 (object 0x00002aaac2df7c50, a
org.apache.activemq.network.DurableConduitBridge),
which is held by k.DurableConduitBridge)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$2.transportResumed(DemandForwardingBridgeSupport.java:220)
at
org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
at
org.apache.activemq.transport.TransportFilter.transportResumed(TransportFilter.java:109)
at
org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:870)
- locked <0x00002aaac2df83c8> (a java.lang.Object)
at
org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:135)
- locked <0x00002aaac2df83d8> (a java.lang.Object)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
