Took a look at the build hangs that have been occurring. I've captured multiple
thread dumps. They all look pretty similar to the following:
"VMTransport" daemon prio=5 tid=0x0000000102384800 nid=0x12011f000 waiting for
monitor entry [0x000000012011e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.activemq.ra.ActiveMQEndpointWorker.getConnection(ActiveMQEndpointWorker.java:281)
- waiting to lock <0x0000000106102000> (a
org.apache.activemq.ra.ActiveMQEndpointWorker$1)
at
org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:109)
at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:135)
at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1485)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:201)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:120)
at
org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:637)
"Default JMS Resource Adapter-worker-1" daemon prio=5 tid=0x000000010236f800
nid=0x11fe16000 in Object.wait() [0x000000011fe15000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000001067a4ba0> (a
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar)
at java.lang.Object.wait(Object.java:485)
at
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75)
- locked <0x00000001067a4ba0> (a
edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar)
at
edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1175)
at
org.apache.activemq.ActiveMQConnectionConsumer.<init>(ActiveMQConnectionConsumer.java:85)
at
org.apache.activemq.ActiveMQConnection.createConnectionConsumer(ActiveMQConnection.java:1085)
at
org.apache.activemq.ra.ActiveMQEndpointWorker$1.run(ActiveMQEndpointWorker.java:166)
- locked <0x0000000106102000> (a
org.apache.activemq.ra.ActiveMQEndpointWorker$1)
at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
at
org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:637)
I believe the "VMTransport" thread is waiting for a lock held by the "Default
JMS Resource Adapter-worker-1". I believe that "Default JMS Resource
Adapter-worker-1" is waiting for a response from the "VMTransport" thread. So,
we're basically in a deadlock.
I only see the problem on AMQ 4.1.2. If I revert back to 4.1.1, I'm not seeing
the problem. I diffed 4.1.1 and 4.1.2, but don't see an obvious cause. Given
that openejb-activemq4 is only in for migration purposes, I think we should
simply revert back to 4.1.1. If anyone is interested in digging into this
further, please feel free!
--kevan
