JMS client failover fails due to a locking issue present in the
Connection/Session classes in the transport package.
--------------------------------------------------------------------------------------------------------------------
Key: QPID-2808
URL: https://issues.apache.org/jira/browse/QPID-2808
Project: Qpid
Issue Type: Bug
Components: Java Client
Reporter: Rajith Attapattu
Assignee: Rajith Attapattu
Fix For: 0.7
Description of problem:
---------------------------------
If the JMS client is connected to a cluster and if the broker the client is
connected to, is killed while the client is in the midst of producing messages,
the client appears to be hung and eventually the failover times out.
The problem only occurs if the producer (or consumer) is using a an addressing
string. If you run the same example using the BURL syntax then there is no
issue.
(*)The resume method in o.a.q.transport.Connection takes a lock when in enters
the resume method and in turn it tries to call the "resumed" methods in the
AMQSession_0_10 class.
(*)That method will essentially recreate the producers and consumers which will
cause the address string to be re-evaluated. In doing so a synchronous
ExchangeQuery is made.
(*) The IoReceiverThread tries to dispatch it, but unable to do as it cannot
grab the 'lock' as it is already held by the failover thread. The failover
thread is not releasing the lock as it is waiting for the response
from the exchange query method.
(*) Hence the client appears to be stuck until the wait for the response times
out and the failover operation fails :)
Below is a partial thread dump that illustrates the above issue.
Following are two snippets from a thread dump that illustrates the issue.
"IoReceiver - /10.16.44.241:5673" daemon prio=10 tid=0x000000000cdf0800
nid=0x1afe waiting for monitor entry [0x0000000041c1a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.qpid.transport.Connection.getSession(Connection.java:440)
- waiting to lock <0x00002aab6075e0e8> (a java.lang.Object)
at org.apache.qpid.transport.Connection.dispatch(Connection.java:380)
at
org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:64)
at
org.apache.qpid.transport.ConnectionDelegate.handle(ConnectionDelegate.java:40)
at
org.apache.qpid.transport.MethodDelegate.sessionAttached(MethodDelegate.java:68)
at org.apache.qpid.transport.SessionAttached.dispatch(SessionAttached.java:91)
at
org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:49)
at
org.apache.qpid.transport.ConnectionDelegate.control(ConnectionDelegate.java:40)
at org.apache.qpid.transport.Method.delegate(Method.java:163)
at org.apache.qpid.transport.Connection.received(Connection.java:348)
at org.apache.qpid.transport.Connection.received(Connection.java:55)
at org.apache.qpid.transport.network.Assembler.emit(Assembler.java:98)
at org.apache.qpid.transport.network.Assembler.assemble(Assembler.java:184)
at org.apache.qpid.transport.network.Assembler.frame(Assembler.java:132)
at org.apache.qpid.transport.network.Frame.delegate(Frame.java:133)
at org.apache.qpid.transport.network.Assembler.received(Assembler.java:103)
at org.apache.qpid.transport.network.Assembler.received(Assembler.java:48)
at org.apache.qpid.transport.network.InputHandler.next(InputHandler.java:187)
at
org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:103)
at
org.apache.qpid.transport.network.InputHandler.received(InputHandler.java:42)
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:128)
at java.lang.Thread.run(Thread.java:636)
IoReceiver - localhost/127.0.0.1:5672" daemon prio=10 tid=0x00002aabbc1a8000
nid=0x1aee in Object.wait() [0x0000000040bee000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab6155be60> (a
org.apache.qpid.transport.Session$ResultFuture)
at org.apache.qpid.transport.util.Waiter.await(Waiter.java:54)
at org.apache.qpid.transport.Session$ResultFuture.get(Session.java:837)
- locked <0x00002aab6155be60> (a
org.apache.qpid.transport.Session$ResultFuture)
at org.apache.qpid.transport.Session$ResultFuture.get(Session.java:859)
at
org.apache.qpid.client.AMQSession_0_10.resolveAddressType(AMQSession_0_10.java:1219)
at
org.apache.qpid.client.AMQSession_0_10.handleAddressBasedDestination(AMQSession_0_10.java:1160)
at
org.apache.qpid.client.BasicMessageProducer_0_10.declareDestination(BasicMessageProducer_0_10.java:81)
at
org.apache.qpid.client.BasicMessageProducer.resubscribe(BasicMessageProducer.java:174)
at
org.apache.qpid.client.AMQSession.resubscribeProducers(AMQSession.java:2782)
at org.apache.qpid.client.AMQSession.resubscribe(AMQSession.java:2038)
at org.apache.qpid.client.AMQSession_0_10.resumed(AMQSession_0_10.java:871)
at org.apache.qpid.transport.Session.resume(Session.java:265)
- locked <0x00002aab61761350> (a [Lorg.apache.qpid.transport.Method;)
at org.apache.qpid.transport.Connection.resume(Connection.java:452)
- locked <0x00002aab6075e0e8> (a java.lang.Object)
at
org.apache.qpid.client.AMQConnectionDelegate_0_10.closed(AMQConnectionDelegate_0_10.java:243)
at org.apache.qpid.transport.Connection.closed(Connection.java:530)
at org.apache.qpid.transport.network.Assembler.closed(Assembler.java:113)
at
org.apache.qpid.transport.network.InputHandler.closed(InputHandler.java:202)
at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:150)
at java.lang.Thread.run(Thread.java:636)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
