I dug up the previous discussion of the problem that I know about... Resending on this thread...
On Feb 23, 2010, at 12:45 PM, Kevan Miller wrote: > > On Feb 23, 2010, at 2:04 AM, David Blevins wrote: > > <snip> > >> Started running into this again and with a little help from Brett Porter >> managed to get some heap dumps. I added the following flag and it redirects >> the output to a file which seems to be enough to get the heap dumps to show >> up somewhere: >> >> mvn clean install -Dmaven.test.redirectTestOutputToFile=true >> >> There's a buffer flushing issue in their "redirect" code, so you have to >> dump twice to build up enough bytes to flush the first dump completely. > > Interesting. Good to know. Thanks. > >> >> Anyway, here's the output. I see a few "locked" items, but it isn't clear >> where the source is. >> > <snip> > >> Found one Java-level deadlock: >> ============================= >> "VMTransport": >> waiting to lock monitor 0x00819694 (object 0x07d11e58, a >> sun.misc.Launcher$AppClassLoader), >> which is held by "main" >> "main": >> waiting to lock monitor 0x008196b8 (object 0x255a4b40, a [[I), >> which is held by "VMTransport" >> >> Java stack information for the threads listed above: >> =================================================== >> "VMTransport": >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:242) >> at >> org.apache.openejb.core.TempClassLoader.loadClass(TempClassLoader.java:92) >> - locked <0x07f2a298> (a org.apache.openejb.core.TempClassLoader) >> at >> org.apache.openejb.core.TempClassLoader.loadClass(TempClassLoader.java:66) >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) >> - locked <0x07f2a298> (a org.apache.openejb.core.TempClassLoader) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:242) >> at >> org.apache.openjpa.enhance.PCClassFileTransformer.needsEnhance(PCClassFileTransformer.java:186) >> at >> org.apache.openjpa.enhance.PCClassFileTransformer.transform0(PCClassFileTransformer.java:132) >> at >> org.apache.openjpa.enhance.PCClassFileTransformer.transform(PCClassFileTransformer.java:120) >> at >> org.apache.openjpa.persistence.PersistenceProviderImpl$ClassTransformerImpl.transform(PersistenceProviderImpl.java:210) >> at >> org.apache.openejb.persistence.PersistenceUnitInfoImpl$PersistenceClassFileTransformer.transform(PersistenceUnitInfoImpl.java:303) >> at >> sun.instrument.TransformerManager.transform(TransformerManager.java:122) >> at >> sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) >> at >> java.util.concurrent.SynchronousQueue$LifoWaitQueue.enq(SynchronousQueue.java:162) >> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:437) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) >> at java.lang.Thread.run(Thread.java:613) >> "main": >> at java.lang.ClassLoader.findBootstrapClass(Native Method) >> at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:308) >> - locked <0x07d14028> (a sun.misc.Launcher$ExtClassLoader) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:306) >> - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) >> - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:251) >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) >> - locked <0x07d11e58> (a sun.misc.Launcher$AppClassLoader) >> at >> org.apache.activemq.ActiveMQConnection.<init>(ActiveMQConnection.java:186) >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:288) >> at >> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:256) >> at >> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:182) >> at >> org.apache.activemq.ra.ActiveMQConnectionSupport.makeConnection(ActiveMQConnectionSupport.java:79) >> at >> org.apache.activemq.ra.ActiveMQConnectionSupport.makeConnection(ActiveMQConnectionSupport.java:61) >> at >> org.apache.activemq.ra.ActiveMQManagedConnectionFactory.createManagedConnection(ActiveMQManagedConnectionFactory.java:169) >> at >> org.apache.geronimo.connector.outbound.MCFConnectionInterceptor.getConnection(MCFConnectionInterceptor.java:48) >> at >> org.apache.geronimo.connector.outbound.XAResourceInsertionInterceptor.getConnection(XAResourceInsertionInterceptor.java:41) >> at >> org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor.internalGetConnection(SinglePoolConnectionInterceptor.java:67) >> - locked <0x07ede778> (a >> org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor$PoolDeque) >> at >> org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:78) >> at >> org.apache.geronimo.connector.outbound.TransactionEnlistingInterceptor.getConnection(TransactionEnlistingInterceptor.java:46) >> at >> org.apache.geronimo.connector.outbound.TransactionCachingInterceptor.getConnection(TransactionCachingInterceptor.java:94) >> at >> org.apache.geronimo.connector.outbound.ConnectionHandleInterceptor.getConnection(ConnectionHandleInterceptor.java:43) >> at >> org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39) >> at >> org.apache.geronimo.connector.outbound.ConnectionTrackingInterceptor.getConnection(ConnectionTrackingInterceptor.java:66) >> at >> org.apache.geronimo.connector.outbound.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:87) >> at >> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:94) >> at >> org.apache.activemq.ra.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:67) >> at >> org.apache.openejb.config.LocalClientTest.assertRefs(LocalClientTest.java:140) >> at >> org.apache.openejb.config.LocalClientTest.test(LocalClientTest.java:118) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:592) >> at junit.framework.TestCase.runTest(TestCase.java:168) >> at junit.framework.TestCase.runBare(TestCase.java:134) >> at junit.framework.TestResult$1.protect(TestResult.java:110) >> at junit.framework.TestResult.runProtected(TestResult.java:128) >> at junit.framework.TestResult.run(TestResult.java:113) >> at junit.framework.TestCase.run(TestCase.java:124) >> at junit.framework.TestSuite.runTest(TestSuite.java:232) >> at junit.framework.TestSuite.run(TestSuite.java:227) >> at >> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:79) >> at >> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:62) >> at >> org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:140) >> at >> org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:165) >> at org.apache.maven.surefire.Surefire.run(Surefire.java:107) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:592) >> at >> org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:289) >> at >> org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:993) >> >> Found 1 deadlock. > > Yuck. This reminds me of a deadlock we had in Geronimo in the JVM's Finalizer > thread. Also, involved a Classloader and a '[[' "object", IIRC. I'll see what > I can remember about that one... The Geronimo Jira was -- https://issues.apache.org/jira/browse/GERONIMO-3687 The Geronimo changes were really work arounds for an underlying JVM problem. IMO, we got lucky that we were able to work around the problem. Here's some indication that the problem has been fixed in the JDK -- http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5088398 -- that's the way to *fix* this problem. --kevan
