[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[
https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646783#action_12646783
]
Shawn Jiang commented on GERONIMO-3687:
---
If you are using WAS CE. I found that this fix had been included in 2.0.0.3
since 2.0.0.1 in this file:
http://publib.boulder.ibm.com/wasce/changes/2003/CHANGES.txt
Additional Apache Geronimo JIRA patches included in WASCE V2.0.0.3:
---
The following Apache Geronimo JIRA patches were included and more
information on each can be found at -
http://issues.apache.org/jira/browse/GERONIMO
4267 - Upgrade to released Yoko 1.0 artifacts
4262 - Include patched build of AMQ 4.1.2 to resolve AMQ-1272 security
exposure
4245 - Upgrade to Tomcat 6.0.18 to pickup latest security fixes
4119 - request.isUserInRole("some-role") always return false after @EJB
injection
4113 - Cannot deploy context specific Valve GBeans on Tomcat
4099 - Calling isUserInRole wipes out run-as info
4096 - Remove PortOffset from SMTPPort definition.
4008 - HTTP GET requests on Provider endpoints might fail sometimes with
Axis2
4006 - Deployment of EJB-based JAX-WS web service with WebServiceContext
injection fails
3965 - Custom LoginModule uses wrong classloader
3930 - IllegalArgumentException reading Transaction Log
3814 - NPE in GBeanOverride.
3687 - classloader deadlock during server startup
> classloader deadlock during server startup
> --
>
> Key: GERONIMO-3687
> URL: https://issues.apache.org/jira/browse/GERONIMO-3687
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
>Affects Versions: 2.0.2, 2.1
>Reporter: Kevan Miller
>Assignee: Kevan Miller
> Fix For: 2.0.3, 2.1.1, 2.2
>
>
> I've been seeing regular deadlocks running Geronimo on Leopard.
> It's the same basic scenario as found in GERONIMO-3141. For some reason, the
> work around in 3141 doesn't work for me on Leopard.
> Just another hack and I'm sure we can fix this...
> Here's background info...
> $ ./geronimo.sh run
> Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2
> Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2
> Using GERONIMO_TMPDIR: var/temp
> Using JRE_HOME:
> /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
> You can then send the java process a QUIT signal (kill -3 ) to have java
> dump out the thread stack traces. The deadlock occurs during a load of an
> Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread
> stack traces that I get:
> Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode):
> "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable
> [0x..0x]
> "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on
> condition [0x..0x]
> "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on
> condition [0x..0x]
> "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor
> entry [0xb0a05000..0xb0a05d90]
> at
> org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35)
> at
> sun.instrument.TransformerManager.transform(TransformerManager.java:122)
> at
> sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155)
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82)
> at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
> "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in
> Object.wait() [0xb0984000..0xb0984d90]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:474)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x05a735f8> (a java.lang.ref.Reference$Lock)
> "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry
> [0xb07ff000..0xb0800188]
> at java.lang.ClassLoader.findBootstrapClass(Native Method)
> at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:308)
> - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280)
> - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646696#action_12646696 ] Kevan Miller commented on GERONIMO-3687: This file on branches/2.0 contains the fix -- https://svn.apache.org/repos/asf/geronimo/server/branches/2.0/modules/geronimo-transformer/src/main/java/org/apache/geronimo/transformer/TransformerCollection.java 2.0.0.1 is not a Geronimo release number. Assume you're using WAS CE... > classloader deadlock during server startup > -- > > Key: GERONIMO-3687 > URL: https://issues.apache.org/jira/browse/GERONIMO-3687 > Project: Geronimo > Issue Type: Bug > Security Level: public(Regular issues) >Affects Versions: 2.0.2, 2.1 >Reporter: Kevan Miller >Assignee: Kevan Miller > Fix For: 2.0.3, 2.1.1, 2.2 > > > I've been seeing regular deadlocks running Geronimo on Leopard. > It's the same basic scenario as found in GERONIMO-3141. For some reason, the > work around in 3141 doesn't work for me on Leopard. > Just another hack and I'm sure we can fix this... > Here's background info... > $ ./geronimo.sh run > Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_TMPDIR: var/temp > Using JRE_HOME: > /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home > You can then send the java process a QUIT signal (kill -3 ) to have java > dump out the thread stack traces. The deadlock occurs during a load of an > Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread > stack traces that I get: > Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode): > "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable > [0x..0x] > "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on > condition [0x..0x] > "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on > condition [0x..0x] > "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor > entry [0xb0a05000..0xb0a05d90] > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at java.lang.ref.Finalizer.access$100(Finalizer.java:14) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) > "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in > Object.wait() [0xb0984000..0xb0984d90] > at java.lang.Object.wait(Native Method) > - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x05a735f8> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry > [0xb07ff000..0xb0800188] > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable > "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on > condition > "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable > Found one Java-level deadlock: > = > "Finalizer": > waiting to lock monitor 0x0081b070 (object 0x05a73660, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I), > which is held by "Finalizer" > Java stack information for the threads listed above: > === > "Finalizer": > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at >
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646672#action_12646672 ] Sree Bodapati commented on GERONIMO-3687: - I am seeing this issue in 2.0.0.1 can someone post a patch on 2.0.0.1 or suggest a way to create this patch so I can avoid this same issue. > classloader deadlock during server startup > -- > > Key: GERONIMO-3687 > URL: https://issues.apache.org/jira/browse/GERONIMO-3687 > Project: Geronimo > Issue Type: Bug > Security Level: public(Regular issues) >Affects Versions: 2.0.2, 2.1 >Reporter: Kevan Miller >Assignee: Kevan Miller > Fix For: 2.0.3, 2.1.1, 2.2 > > > I've been seeing regular deadlocks running Geronimo on Leopard. > It's the same basic scenario as found in GERONIMO-3141. For some reason, the > work around in 3141 doesn't work for me on Leopard. > Just another hack and I'm sure we can fix this... > Here's background info... > $ ./geronimo.sh run > Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_TMPDIR: var/temp > Using JRE_HOME: > /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home > You can then send the java process a QUIT signal (kill -3 ) to have java > dump out the thread stack traces. The deadlock occurs during a load of an > Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread > stack traces that I get: > Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode): > "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable > [0x..0x] > "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on > condition [0x..0x] > "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on > condition [0x..0x] > "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor > entry [0xb0a05000..0xb0a05d90] > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at java.lang.ref.Finalizer.access$100(Finalizer.java:14) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) > "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in > Object.wait() [0xb0984000..0xb0984d90] > at java.lang.Object.wait(Native Method) > - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x05a735f8> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry > [0xb07ff000..0xb0800188] > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable > "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on > condition > "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable > Found one Java-level deadlock: > = > "Finalizer": > waiting to lock monitor 0x0081b070 (object 0x05a73660, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I), > which is held by "Finalizer" > Java stack information for the threads listed above: > === > "Finalizer": > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583693#action_12583693 ] Jarek Gawor commented on GERONIMO-3687: --- Just an update on this problem. I still get this deadlock maybe once per 2 weeks or slightly less. In my case the server is started via geronimo-maven-plugin which polls the server periodically via JMX during start up. So this problem might not be reproduced with geronimo.sh script but might be reproduced with start-server command. > classloader deadlock during server startup > -- > > Key: GERONIMO-3687 > URL: https://issues.apache.org/jira/browse/GERONIMO-3687 > Project: Geronimo > Issue Type: Bug > Security Level: public(Regular issues) >Affects Versions: 2.0.2, 2.1 >Reporter: Kevan Miller >Assignee: Kevan Miller > Fix For: 2.0.x, 2.1.1 > > > I've been seeing regular deadlocks running Geronimo on Leopard. > It's the same basic scenario as found in GERONIMO-3141. For some reason, the > work around in 3141 doesn't work for me on Leopard. > Just another hack and I'm sure we can fix this... > Here's background info... > $ ./geronimo.sh run > Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_TMPDIR: var/temp > Using JRE_HOME: > /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home > You can then send the java process a QUIT signal (kill -3 ) to have java > dump out the thread stack traces. The deadlock occurs during a load of an > Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread > stack traces that I get: > Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode): > "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable > [0x..0x] > "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on > condition [0x..0x] > "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on > condition [0x..0x] > "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor > entry [0xb0a05000..0xb0a05d90] > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at java.lang.ref.Finalizer.access$100(Finalizer.java:14) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) > "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in > Object.wait() [0xb0984000..0xb0984d90] > at java.lang.Object.wait(Native Method) > - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x05a735f8> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry > [0xb07ff000..0xb0800188] > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable > "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on > condition > "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable > Found one Java-level deadlock: > = > "Finalizer": > waiting to lock monitor 0x0081b070 (object 0x05a73660, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I), > which is held by "Finalizer" > Java stack information for the threads listed above: > === > "Finalizer": > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) >
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576586#action_12576586 ] Jarek Gawor commented on GERONIMO-3687: --- This is either a really weird coincidence or something has recently changed but I'm running into this deadlock much more frequently. Just got another one with trunk. The machine is using: java version "1.5.0_12" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04) Java HotSpot(TM) Server VM (build 1.5.0_12-b04, mixed mode) > classloader deadlock during server startup > -- > > Key: GERONIMO-3687 > URL: https://issues.apache.org/jira/browse/GERONIMO-3687 > Project: Geronimo > Issue Type: Bug > Security Level: public(Regular issues) >Affects Versions: 2.0.2, 2.1 >Reporter: Kevan Miller >Assignee: Kevan Miller > Fix For: 2.0.x, 2.1.1 > > > I've been seeing regular deadlocks running Geronimo on Leopard. > It's the same basic scenario as found in GERONIMO-3141. For some reason, the > work around in 3141 doesn't work for me on Leopard. > Just another hack and I'm sure we can fix this... > Here's background info... > $ ./geronimo.sh run > Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_TMPDIR: var/temp > Using JRE_HOME: > /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home > You can then send the java process a QUIT signal (kill -3 ) to have java > dump out the thread stack traces. The deadlock occurs during a load of an > Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread > stack traces that I get: > Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode): > "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable > [0x..0x] > "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on > condition [0x..0x] > "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on > condition [0x..0x] > "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor > entry [0xb0a05000..0xb0a05d90] > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at java.lang.ref.Finalizer.access$100(Finalizer.java:14) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) > "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in > Object.wait() [0xb0984000..0xb0984d90] > at java.lang.Object.wait(Native Method) > - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x05a735f8> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry > [0xb07ff000..0xb0800188] > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable > "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on > condition > "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable > Found one Java-level deadlock: > = > "Finalizer": > waiting to lock monitor 0x0081b070 (object 0x05a73660, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I), > which is held by "Finalizer" > Java stack information for the threads listed above: > === > "Finalizer": > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at >
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576464#action_12576464 ] Kevan Miller commented on GERONIMO-3687: Interesting. Thanks for the info. If we're seeing this on non-Mac, non-Leopard machines prolly time to dig deeper... What JSE level is that on? > classloader deadlock during server startup > -- > > Key: GERONIMO-3687 > URL: https://issues.apache.org/jira/browse/GERONIMO-3687 > Project: Geronimo > Issue Type: Bug > Security Level: public(Regular issues) >Affects Versions: 2.0.2, 2.1 >Reporter: Kevan Miller >Assignee: Kevan Miller > Fix For: 2.0.x, 2.1.1 > > > I've been seeing regular deadlocks running Geronimo on Leopard. > It's the same basic scenario as found in GERONIMO-3141. For some reason, the > work around in 3141 doesn't work for me on Leopard. > Just another hack and I'm sure we can fix this... > Here's background info... > $ ./geronimo.sh run > Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2 > Using GERONIMO_TMPDIR: var/temp > Using JRE_HOME: > /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home > You can then send the java process a QUIT signal (kill -3 ) to have java > dump out the thread stack traces. The deadlock occurs during a load of an > Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread > stack traces that I get: > Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode): > "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable > [0x..0x] > "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on > condition [0x..0x] > "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on > condition [0x..0x] > "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor > entry [0xb0a05000..0xb0a05d90] > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at java.lang.ref.Finalizer.access$100(Finalizer.java:14) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160) > "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in > Object.wait() [0xb0984000..0xb0984d90] > at java.lang.Object.wait(Native Method) > - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:474) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > - locked <0x05a735f8> (a java.lang.ref.Reference$Lock) > "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry > [0xb07ff000..0xb0800188] > at java.lang.ClassLoader.findBootstrapClass(Native Method) > at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946) > at java.lang.ClassLoader.loadClass(ClassLoader.java:308) > - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374) > - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader) > "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable > "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on > condition > "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable > Found one Java-level deadlock: > = > "Finalizer": > waiting to lock monitor 0x0081b070 (object 0x05a73660, a > sun.misc.Launcher$AppClassLoader), > which is held by "main" > "main": > waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I), > which is held by "Finalizer" > Java stack information for the threads listed above: > === > "Finalizer": > at > org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35) > at > sun.instrument.TransformerManager.transform(TransformerManager.java:122) > at > sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) > at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82) > at
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[ https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576229#action_12576229 ] Jarek Gawor commented on GERONIMO-3687: --- I just ran into this deadlock twice in a week on a linux machine: Found one Java-level deadlock: = "RMI TCP Connection(4)-9.42.75.229": waiting to lock monitor 0x081278ec (object 0xd1be5980, a sun.misc.Launcher$AppClassLoader), which is held by "main" "main": waiting to lock monitor 0x08127bac (object 0xc9b84a20, a [[I), which is held by "RMI TCP Connection(4)-9.42.75.229" Java stack information for the threads listed above: === "RMI TCP Connection(4)-9.42.75.229": at java.util.AbstractList.iterator(AbstractList.java:337) at org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:41) at sun.instrument.TransformerManager.transform(TransformerManager.java:122) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) at javax.security.auth.Policy.(Policy.java:145) at javax.security.auth.SubjectDomainCombiner$5.run(SubjectDomainCombiner.java:481) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.SubjectDomainCombiner.compatPolicy(SubjectDomainCombiner.java:477) at javax.security.auth.SubjectDomainCombiner.(SubjectDomainCombiner.java:47) at javax.management.remote.rmi.RMIConnectionImpl.(RMIConnectionImpl.java:124) at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:137) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) at sun.rmi.transport.Transport$1.run(Transport.java:153) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:149) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) at java.lang.Thread.run(Thread.java:595) "main": at java.lang.ClassLoader.findBootstrapClass(Native Method) at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:891) at java.lang.ClassLoader.loadClass(ClassLoader.java:301) - locked <0xd1be8b70> (a sun.misc.Launcher$ExtClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:299) - locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) - locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.findSystemClass(ClassLoader.java:882) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:385) - locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278) - locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) - locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:422) - locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278) - locked
[jira] Commented: (GERONIMO-3687) classloader deadlock during server startup
[
https://issues.apache.org/jira/browse/GERONIMO-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549534
]
Kevan Miller commented on GERONIMO-3687:
>From the stacktrace, the deadlock occurs when we're attempting to load a class
>on line 35 of TransformerCollection:
for (ClassFileTransformer transformer : transformers)
This enhanced for statement is equivalent to:
for (Iterator i = transformer.iterator(); i.hasNext(); ) {
transformer = i.next();
...
}
So, the class being loaded on the Finalizer thread, when the deadlock occurs,
must be an Iterator class. I think if we force a load of the Iterator class
during the load of TransformerCollection, we'll be able to avoid this deadlock.
IMO, the true culprit is the Sun classloader. However, doubt it will be fixed
anytime soon.
Looks like this fixes my problem. I'll commit shortly.
> classloader deadlock during server startup
> --
>
> Key: GERONIMO-3687
> URL: https://issues.apache.org/jira/browse/GERONIMO-3687
> Project: Geronimo
> Issue Type: Bug
> Security Level: public(Regular issues)
>Affects Versions: 2.0.2, 2.1
>Reporter: Kevan Miller
>Assignee: Kevan Miller
> Fix For: 2.0.x, 2.1
>
>
> I've been seeing regular deadlocks running Geronimo on Leopard.
> It's the same basic scenario as found in GERONIMO-3141. For some reason, the
> work around in 3141 doesn't work for me on Leopard.
> Just another hack and I'm sure we can fix this...
> Here's background info...
> $ ./geronimo.sh run
> Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2
> Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2
> Using GERONIMO_TMPDIR: var/temp
> Using JRE_HOME:
> /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
> You can then send the java process a QUIT signal (kill -3 ) to have java
> dump out the thread stack traces. The deadlock occurs during a load of an
> Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread
> stack traces that I get:
> Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode):
> "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable
> [0x..0x]
> "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on
> condition [0x..0x]
> "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on
> condition [0x..0x]
> "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor
> entry [0xb0a05000..0xb0a05d90]
> at
> org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35)
> at
> sun.instrument.TransformerManager.transform(TransformerManager.java:122)
> at
> sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155)
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82)
> at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
> "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in
> Object.wait() [0xb0984000..0xb0984d90]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:474)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x05a735f8> (a java.lang.ref.Reference$Lock)
> "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry
> [0xb07ff000..0xb0800188]
> at java.lang.ClassLoader.findBootstrapClass(Native Method)
> at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:308)
> - locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280)
> - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374)
> - locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
> "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable
> "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on
> condition
> "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable
> Found one Java-level deadlock:
> =
> "Finalizer":
> waiting to lock monitor 0x0081b070 (object 0x05a73660, a
> sun.misc.Launcher$AppClassLoader),
> which is held by "main"
> "main":
> waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I),
>
