Hi, I wrote a test which reproduces the deadlock on Java 6 (MacOS). It's attached to FELIX-4462 ([0]). Unfortunately I only managed to get it fail when running from Eclipse, not via 'mvn clean test'. I had to use byteman to inject a delay in the BundleClassLoader.findClass - and if that inject is not properly applied, the deadlock is not triggered. So there seems to be a difference with byteman-injection under maven vs standalone.
Re a possible fix: I've also added a trivial patch suggesting to synchronized findClass - but fear that this is too heavy, as it would avoid parallel classloading for Java 7++. At least, when synchronizing findClass, my test succeeds :) Cheers, Stefan -- [0] https://issues.apache.org/jira/browse/FELIX-4462 On 3/17/14 4:22 PM, "Richard S. Hall" <[email protected]> wrote: >On 3/17/14, 09:07 , Stefan Egli wrote: >> Hi Richard, >> >> (see below) >> >> On 3/17/14 1:34 PM, "Richard S. Hall" <[email protected]> wrote: >> >>> On 3/14/14, 05:26 , Stefan Egli wrote: >>>> Hi, >>>> >>>> I just ran into the famous deadlock with Java 6 where the same class >>>>is >>>> being loaded from two different bundles (also see FELIX-3953). This >>>> happened on a startup of a CQ instance. >>>> >>>> I'll add more analysis details but my question is: what is the >>>> suggested best practice to handle this issue? >>>> * Use '-XX:+UnlockDiagnosticVMOptions -XX:+UnsyncloadClass' as >>>> suggested in [0] and [1]? But then again [1] says "You must treat it >>>>as >>>> unstable, experimental, temporary, unsupported". >>> This is your only option if you are not able to use Java 7. >>> >>>> * Or would it be worth while to reconsider fixing FELIX-3953? >>> Unless we modify the JRE, there is no way to fix it. The Felix >>>framework >>> doesn't hold locks while class loading. >> I obviously don't know the details of the BundleClassLoader, so I can >>only >> speculate. But it seems the problem is its 'class locking' >>(m_classLocks). >> The assumption there seems, that BundleClassLoader.findClass() is always >> called unsynchronized. But this is clearly not the case in Java 6. > >The underlying issue of eliminating class loader cycle deadlocks is not >possible. > >The m_classLocks structure was put in place to support Java 7 parallel >class loaders. So, I guess the question is, does this approach make >deadlock more likely on older JVMs? I don't know the answer to that nor >do I have the time to look into it, but it would be worthwhile to >investigate/understand. > >> >> So maybe Java 6 would need a special class loader which is >> synchronize-aware.. (in which case you could for example synchronize the >> BundleClassLoaderJava6.findClass() - given that the synchronization >>would >> be done by the VM anyway). >> >> Or introduce a 'proxy class loader' such that the VM would always only >>(be >> able to) synchronize the proxy.. (if that's feasible) > >Again, the underlying issue is not resolvable. Before we did >m_classLocks, we were actually locking on "this" (i.e., the class >loader) before calling findLoadedClass() and that was the only time we >held a lock and we were still getting into deadlocks because of the >JVM-held locks. > >As I state above, the only real issue in my mind is if we were to, for >example, effectively go back to locking on "this" for pre-Java7 JVMs, >would this solve the issue in your specific use case? > >What we were trying to do was to keep a single approach for class >loading, but one that would automatically gain the parallel benefits on >Java 7. It could be possible that it interferes somehow. > >If you want to do a test, modify the class loader code so lock on "this" >instead of m_classLocks when accessing the m_classLocks data structure, >I think that is all that would be necessary to simulate it. If it turns >out to help your situation, then it might be an easy patch. We just have >to modify the class loader to detect which lock it should use. > >-> richard > > >-> richard > >> >> Cheers, >> Stefan >> >>> -> richard >>>> Here's more details (see thread-dumps below): >>>> * Thread 158 (on behalf on bundle X) wants to load class A which is >>>> part of bundle Y - hence goes via getClassByDelegation - then does a >>>> BundleWiringImpl$BundleClassLoader.findClass >>>> * before Thread 158 continues VM threading decides to give Thread >>>>156 >>>> some execution time: >>>> * Thread 156 now wants to load same class A, but this time directly >>>> on behalf of bundle Y. >>>> * Due to [1] Thread 156 internally does a synchronized >>>> loadClassInternal, thus holds the lock on 6f025d710 (the >>>> BundleClassLoader of bundle Y) (I was able to verify this by hooking >>>>the >>>> deadlocked-VM to a debugger, which indicated exactly this) >>>> * Thread 156 now continues into >>>> BundleClassLoader.findClass(BundleWiringImpl.java:2115) where it has >>>>to >>>> do a m_classLocks.wait() - since Thread 158 earlier marked the >>>> BundleClassLoader 'in use/locked for class A'. >>>> * Thread 158 on the other hand continues the findClass execution, >>>> eventually walks into checkCerts, which is synchronized(this), which >>>> Thread 156 holds >>>> * and we have the famous classloading deadlock. >>>> >>>> Cheers, >>>> Stefan >>>> -- >>>> [0] https://issues.apache.org/jira/browse/FELIX-3953 >>>> [1] >>>> >>>>http://underlap.blogspot.de/2006/11/experimental-fix-for-sunbug-4670071 >>>>.h >>>> tml >>>> >>>> PS: Used org.apache.felix.framework 4.3.0.R1558704 - which is a >>>> snapshot-release as of SVN revision 1558704 >>>> >>>> "Thread-158" daemon prio=5 tid=7f83a4cbd800 nid=0x25b73e000 waiting >>>>for >>>> monitor entry [25b73c000] >>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:788) >>>> - waiting to lock <6f025d710> (a >>>> org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5) >>>> at >>>>java.lang.ClassLoader.preDefineClass(ClassLoader.java:493) >>>> at >>>>java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) >>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:621) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass >>>>(B >>>> undleWiringImpl.java:2297) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega >>>>ti >>>> on(BundleWiringImpl.java:1519) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl >>>>.j >>>> ava:77) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass >>>>(B >>>> undleWiringImpl.java:1973) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(Bundle >>>>Wi >>>> ringImpl.java:1392) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.searchImports(BundleWiringI >>>>mp >>>> l.java:1571) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega >>>>ti >>>> on(BundleWiringImpl.java:1502) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl >>>>.j >>>> ava:77) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass >>>>(B >>>> undleWiringImpl.java:1973) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>> [class A being loaded from a class of bundle X] >>>> >>>> "Thread-156" daemon prio=5 tid=7f83a57ac800 nid=0x25b36b000 in >>>> Object.wait() [25b369000] >>>> java.lang.Thread.State: WAITING (on object monitor) >>>> at java.lang.Object.wait(Native Method) >>>> at java.lang.Object.wait(Object.java:485) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.findClass >>>>(B >>>> undleWiringImpl.java:2115) >>>> - locked <6f788bf40> (a java.util.HashMap) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelega >>>>ti >>>> on(BundleWiringImpl.java:1519) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl >>>>.j >>>> ava:77) >>>> at >>>> >>>>org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass >>>>(B >>>> undleWiringImpl.java:1973) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:247) >>>> [class A being loaded from a class of bundle Y] >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: [email protected] >For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
