ActiveMQ non-conformance to JMS Spec causing deadlock when using 3rd-party
Resource Adapter
-------------------------------------------------------------------------------------------
Key: AMQ-2652
URL: https://issues.apache.org/activemq/browse/AMQ-2652
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.3.0
Reporter: Eugene Rodos
Linked to forum discussion:
http://old.nabble.com/ActiveMQ-non-conformance-to-JMS-Spec-causing-deadlock-when-using-3rd-party-Resource-Adapter-tt27869447.html
The following is an excerpt from the JMS Spec 1.1:
{quote}
*8.2.3 ServerSessionPool*
...<snip>...
If the ServerSessionPool is out of ServerSessions, the
getServerSession() method
may block. If a ConnectionConsumer is blocked, it cannot deliver new
messages
until a ServerSession is eventually returned.
{quote}
\\
I am using a 3rd-party JMS Resource Adapter to integrate ActiveMQ with a J2EE
app server. This Resource Adapter does just what the JMS Spec above allows -
it blocks in the ServerSessionPool.getServerSession() call when it is out of
ServerSessions. This causes ActiveMQ to hang due to a deadlock.
The deadlock occurs because the ServerSessionPool uses a single connection to
service all its ServerSessions. This is perfectly legal. However, ActiveMQ is
using this same connection for both dispatching messages to the
ConnectionConsumer as well as communicating transaction commits to the JMS
engine.
So the deadlock scenario is as follows:
\- message 1 is dispatched to the ConnectionConsumer and is being processed
\- before message 1 is committed, message 2 is dispatched
\- ServerSession is requested for processing message 2
\- ServerSessionPool.getServerSession() blocks because it is out of
ServerSessions, blocking the connection from which the call was made
\- message 1 is finished processing and the tx is committed
\- ActiveMQ does a synchronous send using the same connection that is currently
blocked, to communicate the transaction commit command
\- the commit blocks waiting for the connection to be released which will never
happen because that release is dependent on the ServerSession being returned to
the pool which won't happen until the tx is committed
The stacktraces of the 2 deadlocked threads at the time of the deadlock are as
follows:
{noformat}
"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x0aecc9f8
nid=0x1414 in Object.wait() [0x0b6af000..0x0b6afc68]
at java.lang.Object.wait(Native Method)
- waiting on <0x03070188> (a
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool)
at java.lang.Object.wait(Object.java:474)
at
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool.getServerSession(BlockingServerSessionPoolDeadlockTest.java:106)
- locked <0x03070188> (a
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool)
at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:129)
at
org.apache.activemq.ActiveMQConnection$2.processMessageDispatch(ActiveMQConnection.java:1676)
at
org.apache.activemq.command.MessageDispatch.visit(MessageDispatch.java:108)
at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1658)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
- locked <0x03064e98> (a
org.apache.activemq.transport.InactivityMonitor$1)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
at java.lang.Thread.run(Thread.java:595)
"Thread-5" prio=6 tid=0x0af48ae0 nid=0x14a4 waiting on condition
[0x0b8af000..0x0b8afbe8]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1244)
at
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:643)
at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:286)
at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:510)
at
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSession$1.run(BlockingServerSessionPoolDeadlockTest.java:140)
{noformat}
\\
I'm attaching a junit test that illustrates this deadlock.
Now, I realize that this bug does not manifest itself if the ActiveMQ Resource
Adapter is used. However, that is only because of the way AMQ RA works - it
reuses existing ServerSessions instead of blocking when it runs out (and does a
weird callback from the jms session to the ServerSession to start a tx, in
order to avoid delivering several messages on same tx which is what would
happen when reusing an existing ServerSession that is already in the middle of
processing another message).
I think this is a bug in AMQ JMS engine's implementation which just happens to
be masked by the way AMQ RA is implemented. If a Resource Adapted relies on
the blocking behavior clearly provisioned by the JMS Spec, the problem becomes
evident.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
