Re: jetty assembly deadlock (trunk only)

[Kevan Miller]

On Jul 10, 2008, at 10:48 AM, Jarek Gawor wrote:
It seems that once per day one of the automatic trunk builds deadlocks
in the following way (this only happens with the Jetty assembly):

Is anybody else seeing this too? We've had similar problems like  
this before...

I've not seen a deadlock like this, before. The previous ClassLoader  
deadlocks that I recall have involved TransformerCollections and the  
Finalizer thread (these should be fixed).

It looks like AspectJ is breaking the acyclic nature of our DAG  
ClassLoader structure -- causing monitors to be obtained out-of-order  
(making deadlocks a possibility). I haven't located the source for  
org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:73), which appears  
to be the source of the problem. So, don't yet fully understand the  
problem, or how to avoid...

--kevan



Found one Java-level deadlock:
=============================
"RMI TCP Connection(4)-9.42.75.229":
 waiting to lock monitor 0x0808c7fc (object 0xd1418d38, a
org.apache.geronimo.kernel.config.MultiParentClassLoader),
 which is held by "main"
"main":
 waiting to lock monitor 0x0808c83c (object 0xd16967d8, a
org.apache.geronimo.kernel.config.MultiParentClassLoader),
 which is held by "RMI TCP Connection(4)-9.42.75.229"

Java stack information for the threads listed above:
===================================================
"RMI TCP Connection(4)-9.42.75.229":
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadClassInternal(MultiParentClassLoader.java:449)
       - waiting to lock <0xd1418d38> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
       - locked <0xd169c118> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadClassInternal(MultiParentClassLoader.java:459)
       - locked <0xd169c118> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
       - locked <0xd16967d8> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadClassInternal(MultiParentClassLoader.java:459)
       - locked <0xd16967d8> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
       - locked <0xd1530c78> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadOptimizedClass(MultiParentClassLoader.java:410)
       - locked <0xd1530c78> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
281)
       - locked <0xd1530c78> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
319)
       - locked <0xd1530c78> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo.jmxremoting.Authenticator.authenticate(Authenticator.java: 
62)
       at  
javax 
.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java: 
214)
       at  
javax 
.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:181)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at  
sun 
.reflect 
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at  
sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at  
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
       at sun.rmi.transport.Transport$1.run(Transport.java:153)
       at java.security.AccessController.doPrivileged(Native Method)
       at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
       at  
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java: 
466)
       at sun.rmi.transport.tcp.TCPTransport 
$ConnectionHandler.run(TCPTransport.java:707)
       at java.lang.Thread.run(Thread.java:595)
"main":
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadClassInternal(MultiParentClassLoader.java:449)
       - waiting to lock <0xd16967d8> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
       - locked <0xd16aaa28> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadClassInternal(MultiParentClassLoader.java:459)
       - locked <0xd16aaa28> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader.checkParents(MultiParentClassLoader.java:501)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadOptimizedClass(MultiParentClassLoader.java:410)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
281)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
319)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
       at  
java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 
124)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
       at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadOptimizedClass(MultiParentClassLoader.java:425)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
281)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
319)
       - locked <0xd16a1348> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at org.aspectj.weaver.loadtime.Aj 
$WeaverContainer.getWeaver(Aj.java:115)
       - locked <0xd171a720> (a java.util.WeakHashMap)
       at org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:73)
       - locked <0xd1418d38> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.aspectj 
.weaver 
.loadtime 
.ClassPreProcessorAgentAdapter 
.transform(ClassPreProcessorAgentAdapter.java:52)
       at  
org 
.apache 
.geronimo 
.transformer 
.TransformerCollection.transform(TransformerCollection.java:43)
       at  
sun.instrument.TransformerManager.transform(TransformerManager.java: 
122)
       at  
sun 
.instrument.InstrumentationImpl.transform(InstrumentationImpl.java: 
155)
       at java.lang.ClassLoader.defineClass1(Native Method)
       at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
       at  
java.security.SecureClassLoader.defineClass(SecureClassLoader.java: 
124)
       at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
       at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
       at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.MultiParentClassLoader 
.loadOptimizedClass(MultiParentClassLoader.java:425)
       - locked <0xd1418d38> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.kernel 
.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java: 
281)
       - locked <0xd1418d38> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java: 
319)
       - locked <0xd1418d38> (a
org.apache.geronimo.kernel.config.MultiParentClassLoader)
       at  
org 
.apache 
.geronimo 
.common 
.propertyeditor.PropertyEditors.findEditor(PropertyEditors.java:79)
       at  
org 
.apache 
.geronimo 
.common 
.propertyeditor.PropertyEditors.findEditor(PropertyEditors.java:120)
       at  
org 
.apache 
.geronimo 
.system 
.configuration.GBeanOverride.loadPropertyEditor(GBeanOverride.java: 
399)
       at  
org 
.apache 
.geronimo 
.system.configuration.GBeanOverride.getValue(GBeanOverride.java:388)
       - locked <0xd161f8b0> (a
org.apache.geronimo.system.configuration.GBeanOverride)
       at  
org 
.apache 
.geronimo 
.system 
.configuration.GBeanOverride.applyOverrides(GBeanOverride.java:348)
       at  
org 
.apache 
.geronimo 
.system 
.configuration 
.LocalAttributeManager.setAttributes(LocalAttributeManager.java:228)
       - locked <0xd1430c00> (a
org.apache.geronimo.system.configuration.LocalAttributeManager)
       at  
org 
.apache 
.geronimo 
.system 
.configuration 
.LocalAttributeManager.applyOverrides(LocalAttributeManager.java:197)
       - locked <0xd1430c00> (a
org.apache.geronimo.system.configuration.LocalAttributeManager)
       at  
org 
.apache 
.geronimo.kernel.config.Configuration.<init>(Configuration.java:281)
       at  
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at  
sun 
.reflect 
.NativeConstructorAccessorImpl 
.newInstance(NativeConstructorAccessorImpl.java:39)
       at  
sun 
.reflect 
.DelegatingConstructorAccessorImpl 
.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java: 
494)
       at org.apache.xbean.recipe.ReflectionUtil 
$ConstructorFactory.create(ReflectionUtil.java:882)
       at  
org 
.apache.xbean.recipe.ObjectRecipe.internalCreate(ObjectRecipe.java: 
272)
       at  
org.apache.xbean.recipe.AbstractRecipe.create(AbstractRecipe.java:96)
       at  
org.apache.xbean.recipe.AbstractRecipe.create(AbstractRecipe.java:61)
       at  
org 
.apache 
.geronimo 
.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:910)
       at  
org 
.apache 
.geronimo 
.gbean 
.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java: 
269)
       at  
org 
.apache 
.geronimo 
.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:103)
       at  
org 
.apache 
.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:524)
       at  
org 
.apache 
.geronimo.kernel.basic.BasicKernel.startGBean(BasicKernel.java:359)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.KernelConfigurationManager.load(KernelConfigurationManager.java:163)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.SimpleConfigurationManager 
.loadConfiguration(SimpleConfigurationManager.java:312)
       - locked <0xd1677580> (a
org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
       at  
org 
.apache 
.geronimo 
.kernel 
.config 
.SimpleConfigurationManager 
.loadConfiguration(SimpleConfigurationManager.java:280)
       - locked <0xd1677580> (a
org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at  
sun 
.reflect 
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at  
sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at  
org 
.apache 
.geronimo 
.gbean 
.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java: 
34)
       at  
org 
.apache 
.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
       at  
org 
.apache 
.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:815)
       at  
org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java: 
57)
       at  
org 
.apache 
.geronimo 
.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
       at  
org 
.apache 
.geronimo 
.kernel 
.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java: 
96)
       at  
org.apache.geronimo.kernel.config.EditableConfigurationManager$ 
$EnhancerByCGLIB$$aa87e78a.loadConfiguration(<generated>)
       at  
org 
.apache 
.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java: 
158)
       at  
org 
.apache 
.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:78)
       at  
org 
.apache 
.geronimo 
.kernel 
.util 
.MainConfigurationBootstrapper 
.main(MainConfigurationBootstrapper.java:45)
       at  
org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:65)
       at  
org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)

Found 1 deadlock.