Geronimo GBuild uses ActiveMQ to pass build requests/results between GBuild
agents. Twice, now, the M4 ActiveMQ server has become unresponsive and
required a restart. Unfortunately, logging information was not captured.
However, nohup output was captured and contained a thread dump.
I took a look and found two potential problems:
7 threads (the total number of expected clients) were blocked waiting for
memory as follows:
"tcp:///64.14.253.172:23076" daemon prio=1 tid=0xce5014a8 nid=0x3b83 in
Object.wait() [cd25f000..cd260238]
at java.lang.Object.wait(Native Method)
- waiting on <0xd229e7f8> (a java.lang.Object)
at java.lang.Object.wait(Object.java:429)
at org.apache.activemq.memory.UsageManager.waitForSpace(
UsageManager.java:83)
- locked <0xd229e7f8> (a java.lang.Object)
at org.apache.activemq.broker.region.Queue.send(Queue.java:229)
at org.apache.activemq.broker.region.AbstractRegion.send(
AbstractRegion.java:178)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java
:287)
at org.apache.activemq.broker.TransactionBroker.send(
TransactionBroker.java:192)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:94)
at org.apache.activemq.broker.CompositeDestinationBroker.send(
CompositeDestinationBroker.java:97)
at org.apache.activemq.broker.MutableBrokerFilter.send(
MutableBrokerFilter.java:108)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:94)
at org.apache.activemq.broker.AbstractConnection.processMessage(
AbstractConnection.java:332)
at org.apache.activemq.command.ActiveMQMessage.visit(
ActiveMQMessage.java:508)
at org.apache.activemq.broker.AbstractConnection.service(
AbstractConnection.java:185)
at org.apache.activemq.broker.TransportConnection$1.onCommand(
TransportConnection.java:55)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(
ResponseCorrelator.java:82)
at org.apache.activemq.transport.TransportFilter.onCommand(
TransportFilter.java:68)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(
WireFormatNegotiator.java:105)
at org.apache.activemq.transport.TransportSupport.doConsume(
TransportSupport.java:77)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java
:136)
at java.lang.Thread.run(Thread.java:534)
Also, two threads have a suspicious pattern of locks being held/waiting for:
"[EMAIL PROTECTED]://stan.gbuild.org:16161" daemon prio=1
tid=0xced70460 nid=0x3b83 in Object.wait() [cdef8000..cdef9238]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:429)
at edu.emory.mathcs.backport.java.util.concurrent.CountDownLatch.await(
CountDownLatch.java:173)
- locked <0xd269e8b0> (a
edu.emory.mathcs.backport.java.util.concurrent.CountDownLatch)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(
WireFormatNegotiator.java:61)
at org.apache.activemq.transport.MutexTransport.oneway(
MutexTransport.java:44)
- locked <0xd269e8f8> (a java.lang.Object)
at org.apache.activemq.transport.ResponseCorrelator.oneway(
ResponseCorrelator.java:54)
at org.apache.activemq.broker.TransportConnection.dispatch(
TransportConnection.java:189)
at org.apache.activemq.broker.AbstractConnection.start(
AbstractConnection.java:133)
at org.apache.activemq.broker.TransportConnection.start(
TransportConnection.java:70)
at org.apache.activemq.broker.TransportConnector$1.onAccept(
TransportConnector.java:128)
at org.apache.activemq.transport.tcp.TcpTransportServer.run(
TcpTransportServer.java:105)
at java.lang.Thread.run(Thread.java:534)
and
"tcp:///64.14.253.172:21634" daemon prio=1 tid=0x0807cee0 nid=0x3b83 waiting
for monitor entry [cd0dc000..cd0dd238]
at org.apache.activemq.transport.MutexTransport.oneway(
MutexTransport.java:43)
- waiting to lock <0xd269e8f8> (a java.lang.Object)
at org.apache.activemq.transport.ResponseCorrelator.oneway(
ResponseCorrelator.java:54)
at org.apache.activemq.broker.TransportConnection.stop(
TransportConnection.java:76)
at org.apache.activemq.broker.jmx.ManagedTransportConnection.stop(
ManagedTransportConnection.java:62)
at org.apache.activemq.util.ServiceSupport.dispose(ServiceSupport.java
:33)
at
org.apache.activemq.broker.AbstractConnection.serviceTransportException(
AbstractConnection.java:160)
at org.apache.activemq.broker.TransportConnection$1.onException(
TransportConnection.java:61)
at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
at org.apache.activemq.transport.TransportFilter.onException(
TransportFilter.java:102)
at org.apache.activemq.transport.TransportSupport.onException(
TransportSupport.java:90)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java
:149)
at java.lang.Thread.run(Thread.java:534)
--kevan
