Deadlock in ActiveMQ close processing
-------------------------------------
Key: GERONIMO-1493
URL: http://issues.apache.org/jira/browse/GERONIMO-1493
Project: Geronimo
Type: Bug
Components: ActiveMQ
Versions: 1.0
Environment: Geronimo 1.0
Reporter: Kevan Miller
Fix For: 1.1
Poor use of synchronization in ActiveMQAsfEndpointWorker can prevent Geronimo
from shutting down. The problem showed up in Jira 1422
(http://issues.apache.org/jira/browse/GERONIMO-1422). And is contained in the
attached file
(http://issues.apache.org/jira/secure/attachment/12321750/geronimo_shutdown_stdout.txt).
The following thread is attempting to reconnect to the broker:
"Thread-91" prio=7 tid=0x08358d50 nid=0x91 waiting on condition
[c082f000..c082fd98]
at java.lang.Thread.sleep(Native Method)
at
org.activemq.ra.ActiveMQAsfEndpointWorker.reconnect(ActiveMQAsfEndpointWorker.java:177)
- locked <0xc7c80470> (a org.activemq.ra.ActiveMQAsfEndpointWorker)
at
org.activemq.ra.ActiveMQAsfEndpointWorker.access$200(ActiveMQAsfEndpointWorker.java:40)
at
org.activemq.ra.ActiveMQAsfEndpointWorker$1.run(ActiveMQAsfEndpointWorker.java:105)
- locked <0xc7c7d138> (a org.activemq.ra.ActiveMQAsfEndpointWorker$1)
at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:534)
While the following thread is attempting to close the EndpointWorker:
"Geronimo shutdown thread" prio=5 tid=0x08ed3030 nid=0x7f waiting for monitor
entry [c07ed000..c07edd98]
at
org.activemq.ra.ActiveMQAsfEndpointWorker.stop(ActiveMQAsfEndpointWorker.java:135)
- waiting to lock <0xc7c80470> (a
org.activemq.ra.ActiveMQAsfEndpointWorker)
at
org.activemq.ra.ActiveMQResourceAdapter.endpointDeactivation(ActiveMQResourceAdapter.java:261)
at
org.apache.geronimo.connector.ResourceAdapterWrapper.endpointDeactivation(ResourceAdapterWrapper.java:92)
at
org.apache.geronimo.connector.ResourceAdapterWrapper$$FastClassByCGLIB$$4ab28e73.invoke(<generated>)
at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
at
org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
at
org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:118)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:800)
at
org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
at
org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:36)
at
org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
at
org.apache.geronimo.connector.ResourceAdapterWrapper$$EnhancerByCGLIB$$168e117d.endpointDeactivation(<generated>)
at
org.apache.geronimo.connector.ActivationSpecWrapper.deactivate(ActivationSpecWrapper.java:109)
at
org.apache.geronimo.connector.ActivationSpecWrapper$$FastClassByCGLIB$$aaa078c1.invoke(<generated>)
at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
at
org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke(FastMethodInvoker.java:38)
at
org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:118)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:800)
at
org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
at
org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:36)
at
org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
at
org.apache.geronimo.connector.ActivationSpecWrapper$$EnhancerByCGLIB$$17d592bb.deactivate(<generated>)
at org.openejb.mdb.MDBContainer.doStop(MDBContainer.java:223)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.destroyInstance(GBeanInstance.java:1079)
at
org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStop(GBeanInstanceState.java:395)
at
org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:200)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
at
org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
at
org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:192)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
at
org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
at
org.apache.geronimo.gbean.runtime.GBeanInstanceState.stop(GBeanInstanceState.java:192)
at
org.apache.geronimo.gbean.runtime.GBeanInstance.stop(GBeanInstance.java:545)
at
org.apache.geronimo.kernel.basic.BasicKernel.stopGBean(BasicKernel.java:213)
at
org.apache.geronimo.kernel.config.ConfigurationManagerImpl$ShutdownHook.run(ConfigurationManagerImpl.java:287)
at
org.apache.geronimo.kernel.basic.BasicKernel.notifyShutdownHooks(BasicKernel.java:406)
at
org.apache.geronimo.kernel.basic.BasicKernel.shutdown(BasicKernel.java:383)
at org.apache.geronimo.system.main.Daemon$1.run(Daemon.java:272)
ActiveMQAsfEndpointWorker.reconnect is synchronized and is using Thread.sleep
within it's retry loop (not a good idea). Since the reconnect will never
succeed (the broker has been stopped), this prevents Geronimo from ever
closing. I've altered the shutdown ordering. So, it's less likely that the
problem will occur, but the bug still exists...
IMO, the appropriate fix is to use Thread.wait() within reconnect();
maintain/monitor appropriate state (e.g. closed = true) within reconnect() and
close(), and Thread.notify() in close().
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
