On Feb 23, 2010, at 2:04 AM, David Blevins wrote: <snip>
> Started running into this again and with a little help from Brett Porter > managed to get some heap dumps. I added the following flag and it redirects > the output to a file which seems to be enough to get the heap dumps to show > up somewhere: > > mvn clean install -Dmaven.test.redirectTestOutputToFile=true > > There's a buffer flushing issue in their "redirect" code, so you have to dump > twice to build up enough bytes to flush the first dump completely. Interesting. Good to know. Thanks. > > Anyway, here's the output. I see a few "locked" items, but it isn't clear > where the source is. > <snip> > Found one Java-level deadlock: > ============================= > "VMTransport": > waiting to lock monitor 0x00819694 (object 0x07d11e58, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x008196b8 (object 0x255a4b40, a [[I), > which is held by "VMTransport" > > Java stack information for the threads listed above: > =================================================== > "VMTransport": > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:242) > at > org.apache.openejb.core.TempClassLoader.loadClass(TempClassLoader.java:92) > - locked <0x07f2a298> (a org.apache.openejb.core.TempClassLoader) > at > org.apache.openejb.core.TempClassLoader.loadClass(TempClassLoader.java:66) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x07f2a298> (a org.apache.openejb.core.TempClassLoader) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:242) > at > org.apache.openjpa.enhance.PCClassFileTransformer.needsEnhance(PCClassFileTransformer.java:186) > at > org.apache.openjpa.enhance.PCClassFileTransformer.transform0(PCClassFileTransformer.java:132) > at > org.apache.openjpa.enhance.PCClassFileTransformer.transform(PCClassFileTransformer.java:120) > at > org.apache.openjpa.persistence.PersistenceProviderImpl$ClassTransformerImpl.transform(PersistenceProviderImpl.java:210) > at > org.apache.openejb.persistence.PersistenceUnitInfoImpl$PersistenceClassFileTransformer.transform(PersistenceUnitInfoImpl.java:303) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at > java.util.concurrent.SynchronousQueue$LifoWaitQueue.enq(SynchronousQueue.java:162) > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:437) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) > at java.lang.Thread.run(Thread.java:613) > "main": > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x07d14028> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) > at > org.apache.activemq.ActiveMQConnection.<init>(ActiveMQConnection.java:186) > at > org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:288) > at > org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:256) > at > org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:182) > at > org.apache.activemq.ra.ActiveMQConnectionSupport.makeConnection(ActiveMQConnectionSupport.java:79) > at > org.apache.activemq.ra.ActiveMQConnectionSupport.makeConnection(ActiveMQConnectionSupport.java:61) > at > org.apache.activemq.ra.ActiveMQManagedConnectionFactory.createManagedConnection(ActiveMQManagedConnectionFactory.java:169) > at > org.apache.geronimo.connector.outbound.MCFConnectionInterceptor.getConnection(MCFConnectionInterceptor.java:48) > at > org.apache.geronimo.connector.outbound.XAResourceInsertionInterceptor.getConnection(XAResourceInsertionInterceptor.java:41) > at > org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor.internalGetConnection(SinglePoolConnectionInterceptor.java:67) > - locked <0x07ede778> (a > org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor$PoolDeque) > at > org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:78) > at > org.apache.geronimo.connector.outbound.TransactionEnlistingInterceptor.getConnection(TransactionEnlistingInterceptor.java:46) > at > org.apache.geronimo.connector.outbound.TransactionCachingInterceptor.getConnection(TransactionCachingInterceptor.java:94) > at > org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:43) > at > org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39) > at > org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66) > at > org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:87) > at > org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94) > at > org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67) > at > org.apache.openejb.config.LocalClientTest.assertRefs(LocalClientTest.java:140) > at > org.apache.openejb.config.LocalClientTest.test(LocalClientTest.java:118) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:592) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79) > at > org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140) > at > org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165) > at org.apache.maven.surefire.Surefire.run(Surefire.java:107) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:592) > at > org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289) > at > org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:993) > > Found 1 deadlock. Yuck. This reminds me of a deadlock we had in Geronimo in the JVM's Finalizer thread. Also, involved a Classloader and a '[[' "object", IIRC. I'll see what I can remember about that one... --kevan
