RE: RE: [JBoss-dev] Workaround for CL stuff
|VM Spec 8.14 | |Suppose that thread T has in fact performed N lock |operations that have not been matched by unlock |operations. The wait method then adds the current thread |to the wait set for the object, disables the current |thread for thread scheduling purposes, and performs N |unlock operations to relinquish the lock. then we have a fix... marcf | |Regards, |Adrian | |> |AFAIK, a thread will only release the lock in the |> closest synchronization |> |scope. |> |> hence my previous mail |> |> does a double synchronization on the *same* lock |> release both or not. |> |> marcf |> |> | |> |If you can make sure that the UCL itself is the last |> lock before entering |> |ULR, then it |> |should IMHO work. Since we are not in control of |> loadClassInternal, but of |> |loadClass, there is the chance |> |that this will do as wished. |> | |> |CGJ |> | |> |-Ursprüngliche Nachricht- |> |Von: marc fleury [mailto:[EMAIL PROTECTED]] |> |Gesendet: Dienstag, 23. April 2002 03:07 |> |An: Dave Smith |> |Cc: lsanders; Jboss-Development@Lists. Sourceforge. |> Net |> |Betreff: RE: [JBoss-dev] Workaround for CL stuff |> | |> | |> |Solution: |> | |> |When a thread goes through loadClass, in the ULR we |> "lock" with a |> |field that |> |tracks the thread. |> | |> |Another thread comes in, reaches ULR, we sync on the |> calling CL |> |(we know it) |> |we wait, this releases the lock, we keep track of |> the CL as part of this |> |"load". The first thread will always find a CL as |> no CL can be locked for |> |ever, since the thread that would reach the ULR |> would wait. First thread |> |always finishes (with reentrancy so the count must |> go on and we track the |> |fact that it is the same thread). When count |> reaches 0 we get the |> |set of CL |> |that were touched by this thread (i.e. the set of |> CLs that had threads in |> |it) and we notifyAll on each element of the set, |> start this again. |> | |> |Done (afaict), I will try to apply it to the stack |> trace below. |> | |> ||I brought this thing back to life and passed the |> URL of the orginal sun |> ||bug that they rejected. Here is a quick stack trace |> of the deadlock. |> ||Note that you should start jboss with the -Xdebug |> options so it shows |> ||you what objects that it is trying to lock. See |> below for my orginal |> ||post |> || |> || |> ||"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting |> for monitor entry |> ||[0xbb7fe000..0xbb7ffad8] |> ||at |> java.lang.ClassLoader.loadClass(ClassLoader.java:288) |> ||- waiting to lock <43a2c508> (a |> ||org.jboss.mx.loading.UnifiedClassLoader) |> ||at |> ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLoc |> lly(UnifiedCla |> |ssLoader.java:180) |> ||at |> ||org.jboss.mx.loading.UnifiedLoaderRepository.loadCla |> s(UnifiedLoade |> |rRepository.java:178) |> | |> |This one would reach ULR and lock the ULR, see below |> for lock |> |<43a2c508> NOT |> |being locked. |> | |> ||at |> ||org.jboss.mx.loading.UnifiedClassLoader.loadClass(Un |> fiedClassLoade |> ||r.java:217) |> ||at |> java.lang.ClassLoader.loadClass(ClassLoader.java:255) |> ||at |> java.lang.ClassLoader.loadClassInternal(ClassLoader.ja |> a:315) |> ||- locked <43eefa48> (a |> org.jboss.mx.loading.UnifiedClassLoader) |> ||at |> ||com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeS |> ream.java) |> ||at |> ||com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeS |> ream.java) |> ||at |> ||com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7Enc |> deStream.java) |> ||at |> com.candata.gateway.Encryption.Sign(Unknown Source) |> ||- locked <43f0c6a0> (a |> com.candata.gateway.Encryption) |> ||at |> com.candata.gateway.Encryption.createMsg(Unknown |> Source) |> ||at |> com.candata.gateway.CCRAAbstract.postMsg(Unknown |> Source) |> ||at |> com.candata.gateway.CCRAAbstract.recvMsg(Unknown |> Source) |> ||at com.candata.gateway.CCRAPoll.run(Unknown |> Source) |> ||at java.lang.Thread.run(Thread.java:484) |> || |> || |> ||"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting |> for monitor entry |> ||[0xbe7fe000..0xbe7ffad8] |> ||at |> java.lang.ClassLoader.loadClass(ClassLoader.java:288) |> ||- waiting to lock <43eefa48> (a |> ||org.jboss.mx.loading.UnifiedClassLoader) |> ||at |> ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLoc |> lly(UnifiedCla |> |ssLoader.java:180) |> ||at |&
Re: RE: [JBoss-dev] Workaround for CL stuff
VM Spec 8.14 Suppose that thread T has in fact performed N lock operations that have not been matched by unlock operations. The wait method then adds the current thread to the wait set for the object, disables the current thread for thread scheduling purposes, and performs N unlock operations to relinquish the lock. Regards, Adrian > |AFAIK, a thread will only release the lock in the > closest synchronization > |scope. > > hence my previous mail > > does a double synchronization on the *same* lock > release both or not. > > marcf > > | > |If you can make sure that the UCL itself is the last > lock before entering > |ULR, then it > |should IMHO work. Since we are not in control of > loadClassInternal, but of > |loadClass, there is the chance > |that this will do as wished. > | > |CGJ > | > |-Ursprüngliche Nachricht- > |Von: marc fleury [mailto:[EMAIL PROTECTED]] > |Gesendet: Dienstag, 23. April 2002 03:07 > |An: Dave Smith > |Cc: lsanders; Jboss-Development@Lists. Sourceforge. > Net > |Betreff: RE: [JBoss-dev] Workaround for CL stuff > | > | > |Solution: > | > |When a thread goes through loadClass, in the ULR we > "lock" with a > |field that > |tracks the thread. > | > |Another thread comes in, reaches ULR, we sync on the > calling CL > |(we know it) > |we wait, this releases the lock, we keep track of > the CL as part of this > |"load". The first thread will always find a CL as > no CL can be locked for > |ever, since the thread that would reach the ULR > would wait. First thread > |always finishes (with reentrancy so the count must > go on and we track the > |fact that it is the same thread). When count > reaches 0 we get the > |set of CL > |that were touched by this thread (i.e. the set of > CLs that had threads in > |it) and we notifyAll on each element of the set, > start this again. > | > |Done (afaict), I will try to apply it to the stack > trace below. > | > ||I brought this thing back to life and passed the > URL of the orginal sun > ||bug that they rejected. Here is a quick stack trace > of the deadlock. > ||Note that you should start jboss with the -Xdebug > options so it shows > ||you what objects that it is trying to lock. See > below for my orginal > ||post > || > || > ||"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting > for monitor entry > ||[0xbb7fe000..0xbb7ffad8] > ||at > java.lang.ClassLoader.loadClass(ClassLoader.java:288) > ||- waiting to lock <43a2c508> (a > ||org.jboss.mx.loading.UnifiedClassLoader) > ||at > ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLoc > lly(UnifiedCla > |ssLoader.java:180) > ||at > ||org.jboss.mx.loading.UnifiedLoaderRepository.loadCla > s(UnifiedLoade > |rRepository.java:178) > | > |This one would reach ULR and lock the ULR, see below > for lock > |<43a2c508> NOT > |being locked. > | > ||at > ||org.jboss.mx.loading.UnifiedClassLoader.loadClass(Un > fiedClassLoade > ||r.java:217) > ||at > java.lang.ClassLoader.loadClass(ClassLoader.java:255) > ||at > java.lang.ClassLoader.loadClassInternal(ClassLoader.ja > a:315) > ||- locked <43eefa48> (a > org.jboss.mx.loading.UnifiedClassLoader) > ||at > ||com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeS > ream.java) > ||at > ||com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeS > ream.java) > ||at > ||com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7Enc > deStream.java) > ||at > com.candata.gateway.Encryption.Sign(Unknown Source) > ||- locked <43f0c6a0> (a > com.candata.gateway.Encryption) > ||at > com.candata.gateway.Encryption.createMsg(Unknown > Source) > ||at > com.candata.gateway.CCRAAbstract.postMsg(Unknown > Source) > ||at > com.candata.gateway.CCRAAbstract.recvMsg(Unknown > Source) > ||at com.candata.gateway.CCRAPoll.run(Unknown > Source) > ||at java.lang.Thread.run(Thread.java:484) > || > || > ||"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting > for monitor entry > ||[0xbe7fe000..0xbe7ffad8] > ||at > java.lang.ClassLoader.loadClass(ClassLoader.java:288) > ||- waiting to lock <43eefa48> (a > ||org.jboss.mx.loading.UnifiedClassLoader) > ||at > ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLoc > lly(UnifiedCla > |ssLoader.java:180) > ||at > ||org.jboss.mx.loading.UnifiedLoaderRepository.loadCla > s(UnifiedLoade > |rRepository.java:178) > | > |thread 2 reaches here and sees a ULR under usage. It >
RE: [JBoss-dev] Workaround for CL stuff
|AFAIK, a thread will only release the lock in the closest synchronization |scope. hence my previous mail does a double synchronization on the *same* lock release both or not. marcf | |If you can make sure that the UCL itself is the last lock before entering |ULR, then it |should IMHO work. Since we are not in control of loadClassInternal, but of |loadClass, there is the chance |that this will do as wished. | |CGJ | |-Ursprüngliche Nachricht- |Von: marc fleury [mailto:[EMAIL PROTECTED]] |Gesendet: Dienstag, 23. April 2002 03:07 |An: Dave Smith |Cc: lsanders; Jboss-Development@Lists. Sourceforge. Net |Betreff: RE: [JBoss-dev] Workaround for CL stuff | | |Solution: | |When a thread goes through loadClass, in the ULR we "lock" with a |field that |tracks the thread. | |Another thread comes in, reaches ULR, we sync on the calling CL |(we know it) |we wait, this releases the lock, we keep track of the CL as part of this |"load". The first thread will always find a CL as no CL can be locked for |ever, since the thread that would reach the ULR would wait. First thread |always finishes (with reentrancy so the count must go on and we track the |fact that it is the same thread). When count reaches 0 we get the |set of CL |that were touched by this thread (i.e. the set of CLs that had threads in |it) and we notifyAll on each element of the set, start this again. | |Done (afaict), I will try to apply it to the stack trace below. | ||I brought this thing back to life and passed the URL of the orginal sun ||bug that they rejected. Here is a quick stack trace of the deadlock. ||Note that you should start jboss with the -Xdebug options so it shows ||you what objects that it is trying to lock. See below for my orginal ||post || || ||"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting for monitor entry ||[0xbb7fe000..0xbb7ffad8] ||at java.lang.ClassLoader.loadClass(ClassLoader.java:288) ||- waiting to lock <43a2c508> (a ||org.jboss.mx.loading.UnifiedClassLoader) ||at ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla |ssLoader.java:180) ||at ||org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade |rRepository.java:178) | |This one would reach ULR and lock the ULR, see below for lock |<43a2c508> NOT |being locked. | ||at ||org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade ||r.java:217) ||at java.lang.ClassLoader.loadClass(ClassLoader.java:255) ||at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) ||- locked <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) ||at ||com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeStream.java) ||at ||com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeStream.java) ||at ||com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7EncodeStream.java) ||at com.candata.gateway.Encryption.Sign(Unknown Source) ||- locked <43f0c6a0> (a com.candata.gateway.Encryption) ||at com.candata.gateway.Encryption.createMsg(Unknown Source) ||at com.candata.gateway.CCRAAbstract.postMsg(Unknown Source) ||at com.candata.gateway.CCRAAbstract.recvMsg(Unknown Source) ||at com.candata.gateway.CCRAPoll.run(Unknown Source) ||at java.lang.Thread.run(Thread.java:484) || || ||"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting for monitor entry ||[0xbe7fe000..0xbe7ffad8] ||at java.lang.ClassLoader.loadClass(ClassLoader.java:288) ||- waiting to lock <43eefa48> (a ||org.jboss.mx.loading.UnifiedClassLoader) ||at ||org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla |ssLoader.java:180) ||at ||org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade |rRepository.java:178) | |thread 2 reaches here and sees a ULR under usage. It waits on the calling |UCL (which we know it is passed) and THEREFORE UNLOCKS <43a2c508>. | |Done | |marcf | ||at ||org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade ||r.java:217) | ||at java.lang.ClassLoader.loadClass(ClassLoader.java:255) ||at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) ||- locked <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) ||at java.lang.Class.forName0(Native Method) ||at java.lang.Class.forName(Class.java:195) ||at ||javax.security.auth.login.LoginContext.invoke(LoginContext.java:626) ||at ||javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) ||at ||javax.security.auth.login.LoginContext$4.run(LoginContext.java:599) ||at java.security.AccessController.doPrivileged(Native Method) ||at ||javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:596) ||at ||javax.security.auth.login.LoginContext.login(LoginContext.java:523) ||
AW: [JBoss-dev] Workaround for CL stuff
AFAIK, a thread will only release the lock in the closest synchronization scope. If you can make sure that the UCL itself is the last lock before entering ULR, then it should IMHO work. Since we are not in control of loadClassInternal, but of loadClass, there is the chance that this will do as wished. CGJ -Ursprüngliche Nachricht- Von: marc fleury [mailto:[EMAIL PROTECTED]] Gesendet: Dienstag, 23. April 2002 03:07 An: Dave Smith Cc: lsanders; Jboss-Development@Lists. Sourceforge. Net Betreff: RE: [JBoss-dev] Workaround for CL stuff Solution: When a thread goes through loadClass, in the ULR we "lock" with a field that tracks the thread. Another thread comes in, reaches ULR, we sync on the calling CL (we know it) we wait, this releases the lock, we keep track of the CL as part of this "load". The first thread will always find a CL as no CL can be locked for ever, since the thread that would reach the ULR would wait. First thread always finishes (with reentrancy so the count must go on and we track the fact that it is the same thread). When count reaches 0 we get the set of CL that were touched by this thread (i.e. the set of CLs that had threads in it) and we notifyAll on each element of the set, start this again. Done (afaict), I will try to apply it to the stack trace below. |I brought this thing back to life and passed the URL of the orginal sun |bug that they rejected. Here is a quick stack trace of the deadlock. |Note that you should start jboss with the -Xdebug options so it shows |you what objects that it is trying to lock. See below for my orginal |post | | |"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting for monitor entry |[0xbb7fe000..0xbb7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43a2c508> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) This one would reach ULR and lock the ULR, see below for lock <43a2c508> NOT being locked. |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) |at |com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7EncodeStream.java) |at com.candata.gateway.Encryption.Sign(Unknown Source) |- locked <43f0c6a0> (a com.candata.gateway.Encryption) |at com.candata.gateway.Encryption.createMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.postMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.recvMsg(Unknown Source) |at com.candata.gateway.CCRAPoll.run(Unknown Source) |at java.lang.Thread.run(Thread.java:484) | | |"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting for monitor entry |[0xbe7fe000..0xbe7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43eefa48> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) thread 2 reaches here and sees a ULR under usage. It waits on the calling UCL (which we know it is passed) and THEREFORE UNLOCKS <43a2c508>. Done marcf |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) |at java.lang.Class.forName0(Native Method) |at java.lang.Class.forName(Class.java:195) |at |javax.security.auth.login.LoginContext.invoke(LoginContext.java:626) |at |javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) |at |javax.security.auth.login.LoginContext$4.run(LoginContext.java:599) |at java.security.AccessController.doPrivileged(Native Method) |at |javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:596) |at |javax.security.auth.login.LoginContext.login(LoginContext.java:523) |at com.candata.util.beans.CandataClientLogin.login(Unknown |Source) |at com.candata.bbxinterface.BbxJavaInvoker.login(Unknown Source) |at com.candata.bbxinterface.BbxWrapper.run(Unknown Source) |at java.lang.Thread.r
AW: [JBoss-dev] Workaround for CL stuff
Actually, it´s more complicated: Thread 1: VM detects that it needs bar, calls UCL A.loadClassInternal() and hence synchronizes on A Thread 2: VM detects that it needs foo, calls UCL B.loadClassInternal() and hence synchronizes on B Thread 1: UCL A delegates to Repository delegates to UCL B, LOCKED Thread 2: UCL B delegates to Repository delegates to UCL A, LOCKED CGJ -Ursprüngliche Nachricht- Von: lsanders [mailto:[EMAIL PROTECTED]] Gesendet: Montag, 22. April 2002 20:29 An: Jboss-Development@Lists. Sourceforge. Net Betreff: Re: [JBoss-dev] Workaround for CL stuff Though I've never experienced this, I think this is the problem scenario: The players: UnifiedClassLoader A (can load directly class foo) UnifiedClassLoader B (can load directly class bar) Thread 1 (context loader is A): Load new class bar: - synchronize on UCL A - Search UCL B - Attempt to synchronize on UCL B Thread 2 (contect loader is B): Load new class foo: - synchronize on UCL B - Search UCL A - Attempt to synchronize on UCL A Can someone verify if this is accurate? -Larry > Ok, > > I am sure we can find something. > > I would appreciate a brief description of a CL deadlock scenario due > to the > final loadClassInternal. Jung? Sacha? > > It's only software, software is dumb > > marcf > > x > Marc Fleury, Ph.D > President and CEO > JBoss Group, LLC > x > > > > ___ > Jboss-development mailing list [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/jboss-development > ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Solution: When a thread goes through loadClass, in the ULR we "lock" with a field that tracks the thread. Another thread comes in, reaches ULR, we sync on the calling CL (we know it) we wait, this releases the lock, we keep track of the CL as part of this "load". The first thread will always find a CL as no CL can be locked for ever, since the thread that would reach the ULR would wait. First thread always finishes (with reentrancy so the count must go on and we track the fact that it is the same thread). When count reaches 0 we get the set of CL that were touched by this thread (i.e. the set of CLs that had threads in it) and we notifyAll on each element of the set, start this again. Done (afaict), I will try to apply it to the stack trace below. |I brought this thing back to life and passed the URL of the orginal sun |bug that they rejected. Here is a quick stack trace of the deadlock. |Note that you should start jboss with the -Xdebug options so it shows |you what objects that it is trying to lock. See below for my orginal |post | | |"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting for monitor entry |[0xbb7fe000..0xbb7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43a2c508> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) This one would reach ULR and lock the ULR, see below for lock <43a2c508> NOT being locked. |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) |at |com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7EncodeStream.java) |at com.candata.gateway.Encryption.Sign(Unknown Source) |- locked <43f0c6a0> (a com.candata.gateway.Encryption) |at com.candata.gateway.Encryption.createMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.postMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.recvMsg(Unknown Source) |at com.candata.gateway.CCRAPoll.run(Unknown Source) |at java.lang.Thread.run(Thread.java:484) | | |"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting for monitor entry |[0xbe7fe000..0xbe7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43eefa48> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) thread 2 reaches here and sees a ULR under usage. It waits on the calling UCL (which we know it is passed) and THEREFORE UNLOCKS <43a2c508>. Done marcf |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) |at java.lang.Class.forName0(Native Method) |at java.lang.Class.forName(Class.java:195) |at |javax.security.auth.login.LoginContext.invoke(LoginContext.java:626) |at |javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) |at |javax.security.auth.login.LoginContext$4.run(LoginContext.java:599) |at java.security.AccessController.doPrivileged(Native Method) |at |javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:596) |at |javax.security.auth.login.LoginContext.login(LoginContext.java:523) |at com.candata.util.beans.CandataClientLogin.login(Unknown |Source) |at com.candata.bbxinterface.BbxJavaInvoker.login(Unknown Source) |at com.candata.bbxinterface.BbxWrapper.run(Unknown Source) |at java.lang.Thread.run(Thread.java:484) marcf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
|Also ClassLoader.loadClass(String, boolean) is sync'ed. It's not a
|problem of loadClassInternal IMHO.
|Also the UCL at the end calls super.loadClass() (in
|loadClassLocally()) so...
right so it is just a problem with synchronized cls...
I will try the solution I am thinking of.
hey quick question
synchronized(a)
{
synchronized (a)
{
a.wait(blabla);
}}
does it release a?
I think so
(if one is b I know it doesn't)
|Not only. It's a startup problem IMHO: until all classes are
|cached by ULR, you may find that UCL calls super.loadClass() and
|you have the same problem.
ok, which also means that if all threads go through our stuff we can solve
all of this in one central place.
|It is sync'ed, and would lead to the same problem, no ?
yes afaict, and the solution would apply to both.
|I agree that sync'ing so high it's an implementation mistake, they
|could have sync'ed around shared data structures, as ULR does.
marcf
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
I'm not very knowledgable when it comes to Classloaders, so please
accept my apologies if this idea is naïve.
But why couldn't you factor the code that knows how to search the Jars
(or whatever) that each respective classloader is responsible for out
into a separate object. Then instead of Classloader A calling
Classloader B to load a class on its behalf (and hence causing a
deadlock, it would instead call a method on Classloader B's delegate
object).
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] On Behalf Of marc
fleury
Sent: Monday, April 22, 2002 5:47 PM
To: David Maplesden; Jboss-Development@Lists. Sourceforge. Net
Subject: RE: [JBoss-dev] Workaround for CL stuff
Ok...
I think I see,
but then there might be a solution which is to sleep one of the thread
which will release the lock for that we need detection at the ULR level
of usage wich we can probably do... hmm interesting.
I need to go to a party at my kids school and will think about this...
thanks for the input guys
marcf
|-Original Message-
|From: [EMAIL PROTECTED]
|[mailto:[EMAIL PROTECTED]]On Behalf Of
|David Maplesden
|Sent: Monday, April 22, 2002 2:36 PM
|To: 'marc fleury'; Jboss-Development@Lists. Sourceforge. Net
|Subject: RE: [JBoss-dev] Workaround for CL stuff
|
|
|Ok, I'm not the expert on this stuff either, but in an effort to help
|out here is my explanation of what I think is happening.
|
|The major problem (and the starting point of the whole thing) is that
|when classes are being loaded the JVM can call "loadClassInternal()" on
|multiple class loaders at the same time. This method is synchronized
|and so immediately two different threads have locked two different
|class loaders, let us say Thread A has locked ClassLoader A and Thread
|B has locked ClassLoader B.
|
|Now loadClassInternal simply calls the loadClass method of the
|respective class loader and so if you have a class loader (like the
|JBoss
|UnifiedClassLoader) which can delegate to other class loaders (via
|UnifiedLoaderRepository) to load classes you can get a situation where
|Thread A now wants to access synchronized methods in ClassLoader B and
|Thread B wants to access synchronized methods in ClassLoader A and
neither
|can because of the earlier locks already gained, hence deadlock.
|
|This bought about by the fact that the private "loadClassInternal()"
|method is synchronized and can be called pretty much at any time by the
|JVM.
|
|The only way you could work around the problem is if you could
|synchronize on some common object before loadClassInternal is called,
|but since this is called rather unpredictably directly from the JVM I
|don't know if there is any way to do this.
|
|Why the F*K "loadClassInternal()" is synchronized is a complete mystery
|as all it does is call "loadClass()" which can be synchronized if it
|needs to be.
|
|David
|
|
|> -Original Message-
|> From: marc fleury [mailto:[EMAIL PROTECTED]]
|> Sent: Tuesday, April 23, 2002 8:47 AM
|> To: Bordet, Simone; Jboss-Development@Lists. Sourceforge. Net
|> Subject: RE: [JBoss-dev] Workaround for CL stuff
|>
|>
|> Simone,
|>
|> I still don't see the problem clearly, please help me.
|>
|> |> Where does the "loadClassInternal()" intervene?
|> |
|> |It is called by the JVM when it has to resolve class dependencies.
|>
|> ok
|>
|> |Say you have class A that has a data member of class B. When a CL is
|> |asked to load A, it inspects the class and see that it needs to load
|> |also B. This is done at native level, and while a classloader is
|> |loading class A, the JVM calls loadClassInternal() on the *same*
|> |thread and on the *same* classloader to load class B; this may
|> |trigger another call and so on recursively. If the
|>
|> ok
|>
|> |classloader delegates to some other classloader you can have
|> deadlock.
|>
|> this I don't see, you seem to jump to the conclusion, can you please
|> walk me?
|>
|> |There is a call for loadClass, some more call on java code, then a
|> |call to native method, and if the native method decides it has no
|> |sufficient information it calls back java code, it's java -> native
|> |-> java.
|> |
|> |loadClass()
|> | ...
|> |defineClass() [is this the native one ?]
|> | loadClassInternal()
|> |loadClass()
|>
|> So if loadClassInternal fails it goes to the "loadClass()" in our
|> stack?
|>
|> cl1.loadClass("A")
|> A refers to B
|> cl1.loadClassInternal("B") fails
|> cl1.loadClass("B") is then called.
|>
|> is that correct?
|>
|> And then ...?
|>
|> |A typical scenario for me was this one: jar1 with a set of related
|> |classes, and jar2 with another set
RE: [JBoss-dev] Workaround for CL stuff
|You have 2 threads, both waiting for something, just started, so
|not classes have been loaded yet. (see Dave Smith post for stack
|trace and example).
|Thread "CCRAPoll" (call it Encryption thread) asks the CtxCL to
|load something. CtxCL is some JBoss UCL.
|Other thread is "Thread-20" (call it Login thread) again uses the CtxCL.
|Both CLs don't know the class and delegate to the
|UnifiedRepository which iterates on the CLs asking.
|Same does the other thread. If you are unlucky, deadlock. Can you
|see it now ?
yes
|Say you have only 2 CLs
|thread1 does: CL1.load("A") -> ULR.load("A"), CL1.load("A"), can't
|find it, iterates on CLs (only CL2) -> CL2.load("A");
|thread2 does: CL2.load("B") -> ULR.load("B"), CL2.load("B"), can't
|find it, iterates on CLs (only CL1) -> CL1.load("B");
yes, if we do lock on the cl itself (I don't remember) then yes.
|Imagine the first load is triggered by loadClassInternal() (see
|Encryption thread, PKCS7EncodeStream.e() have to load some class,
|and loadClassInternal is called, this syncs a JBoss UCL); the
|second load calls UCL.loadClassLocally() which calls super, which
|is sync'ed as well :) Bang ! See it now ?
yes
|Thinking loud for a solution:
|
|1- Resource ordering may help ? I mean if you don't iterate on CL
|in ULR, but first you order them, then loop ? Not sure it will
|work in all cases...
|2- What if you don't call super() which is sync'ed and you make
|other 2 UCL, one that "wraps" jre/lib/ext and one that "wraps" the
|classpath, both with parent the boot CL (ie null) ? Don't know if
|the boot classloader is sync'ed somewhere in native code... Make
|the flat classloader space in orbit around the boot classloader,
|not the system classloader...
|3- The more I think the more it seems to me that if you sync all
|CL in the correct order you can get out...
|Kind of:
|
|sync(cl1) {
| sync(cl2) {
|sync(cl3) {
| ..
|sync(cln) {
| cln.loadClass(..)
|}
| }
|}
| }
|}
|
|where cl1 < cl2 < cl3 < ... < cln and there is a clearly defined
|"operator <" between classloaders (hashcode may help, is it unique
|when we talk of Object.hashCode() ?)
|
|Of course since you don't know a priori the number of classloaders
|you have to use some fancy Doug Lea's classes :)
|
|Good coding, I know you like this stuff :)
I think I have a solution just make it for a single thread model, and the
only tricky thing (which I was thinking about earlier) would be to track if
someone is using a given CL... i think it will be easy to code, I have to do
it.
I will try it on the 3.1 branch with backporting to 3.0
Thanks for the help simone
marcf
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Hi, > The major problem (and the starting point of the whole thing) > is that when > classes are being loaded the JVM can call > "loadClassInternal()" on multiple > class loaders at the same time. This method is synchronized and so > immediately two different threads have locked two different > class loaders, > let us say Thread A has locked ClassLoader A and Thread B has locked > ClassLoader B. Also ClassLoader.loadClass(String, boolean) is sync'ed. It's not a problem of loadClassInternal IMHO. Also the UCL at the end calls super.loadClass() (in loadClassLocally()) so... > Now loadClassInternal simply calls the loadClass method of > the respective > class loader and so if you have a class loader (like the JBoss > UnifiedClassLoader) which can delegate to other class loaders (via > UnifiedLoaderRepository) to load classes you can get a situation where > Thread A now wants to access synchronized methods in ClassLoader B and > Thread B wants to access synchronized methods in ClassLoader > A and neither > can because of the earlier locks already gained, hence deadlock. Correct. > This bought about by the fact that the private > "loadClassInternal()" method > is synchronized and can be called pretty much at any time by > the JVM. Not only. It's a startup problem IMHO: until all classes are cached by ULR, you may find that UCL calls super.loadClass() and you have the same problem. > The only way you could work around the problem is if you > could synchronize > on some common object before loadClassInternal is called, but > since this is > called rather unpredictably directly from the JVM I don't > know if there is > any way to do this. > > Why the F*K "loadClassInternal()" is synchronized is a > complete mystery as > all it does is call "loadClass()" which can be synchronized > if it needs to > be. It is sync'ed, and would lead to the same problem, no ? I *think* (but not sure) that loadClass is sync'ed to ensure correct class initialization. If 2 threads go step-by-step in parallel to load the same class with the same CL, somewhere you have to sync to initialize the class only once... I agree that sync'ing so high it's an implementation mistake, they could have sync'ed around shared data structures, as ULR does. Simon ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Hi Marc,
> Simone,
>
> I still don't see the problem clearly, please help me.
Will try to do my best. I used the now "old" (thanks to you) classloader delegation
model in my project, so you probably have to integrate me with the JBoss stuff
(correct me if I'm wrong) :)
> |classloader delegates to some other classloader you can have
> deadlock.
>
> this I don't see, you seem to jump to the conclusion, can you
> please walk me?
You have 2 threads, both waiting for something, just started, so not classes have been
loaded yet. (see Dave Smith post for stack trace and example).
Thread "CCRAPoll" (call it Encryption thread) asks the CtxCL to load something. CtxCL
is some JBoss UCL.
Other thread is "Thread-20" (call it Login thread) again uses the CtxCL.
Both CLs don't know the class and delegate to the UnifiedRepository which iterates on
the CLs asking.
Same does the other thread. If you are unlucky, deadlock. Can you see it now ?
Say you have only 2 CLs
thread1 does: CL1.load("A") -> ULR.load("A"), CL1.load("A"), can't find it, iterates
on CLs (only CL2) -> CL2.load("A");
thread2 does: CL2.load("B") -> ULR.load("B"), CL2.load("B"), can't find it, iterates
on CLs (only CL1) -> CL1.load("B");
Imagine the first load is triggered by loadClassInternal() (see Encryption thread,
PKCS7EncodeStream.e() have to load some class, and loadClassInternal is called, this
syncs a JBoss UCL); the second load calls UCL.loadClassLocally() which calls super,
which is sync'ed as well :) Bang ! See it now ?
Thinking loud for a solution:
1- Resource ordering may help ? I mean if you don't iterate on CL in ULR, but first
you order them, then loop ? Not sure it will work in all cases...
2- What if you don't call super() which is sync'ed and you make other 2 UCL, one that
"wraps" jre/lib/ext and one that "wraps" the classpath, both with parent the boot CL
(ie null) ? Don't know if the boot classloader is sync'ed somewhere in native code...
Make the flat classloader space in orbit around the boot classloader, not the system
classloader...
3- The more I think the more it seems to me that if you sync all CL in the correct
order you can get out...
Kind of:
sync(cl1) {
sync(cl2) {
sync(cl3) {
..
sync(cln) {
cln.loadClass(..)
}
}
}
}
}
where cl1 < cl2 < cl3 < ... < cln and there is a clearly defined "operator <" between
classloaders (hashcode may help, is it unique when we talk of Object.hashCode() ?)
Of course since you don't know a priori the number of classloaders you have to use
some fancy Doug Lea's classes :)
Good coding, I know you like this stuff :)
Simon
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Ok...
I think I see,
but then there might be a solution which is to sleep one of the thread which
will release the lock for that we need detection at the ULR level of usage
wich we can probably do... hmm interesting.
I need to go to a party at my kids school and will think about this...
thanks for the input guys
marcf
|-Original Message-
|From: [EMAIL PROTECTED]
|[mailto:[EMAIL PROTECTED]]On Behalf Of David
|Maplesden
|Sent: Monday, April 22, 2002 2:36 PM
|To: 'marc fleury'; Jboss-Development@Lists. Sourceforge. Net
|Subject: RE: [JBoss-dev] Workaround for CL stuff
|
|
|Ok, I'm not the expert on this stuff either, but in an effort to help out
|here is my explanation of what I think is happening.
|
|The major problem (and the starting point of the whole thing) is that when
|classes are being loaded the JVM can call "loadClassInternal()" on multiple
|class loaders at the same time. This method is synchronized and so
|immediately two different threads have locked two different class loaders,
|let us say Thread A has locked ClassLoader A and Thread B has locked
|ClassLoader B.
|
|Now loadClassInternal simply calls the loadClass method of the respective
|class loader and so if you have a class loader (like the JBoss
|UnifiedClassLoader) which can delegate to other class loaders (via
|UnifiedLoaderRepository) to load classes you can get a situation where
|Thread A now wants to access synchronized methods in ClassLoader B and
|Thread B wants to access synchronized methods in ClassLoader A and neither
|can because of the earlier locks already gained, hence deadlock.
|
|This bought about by the fact that the private "loadClassInternal()" method
|is synchronized and can be called pretty much at any time by the JVM.
|
|The only way you could work around the problem is if you could synchronize
|on some common object before loadClassInternal is called, but since this is
|called rather unpredictably directly from the JVM I don't know if there is
|any way to do this.
|
|Why the F*K "loadClassInternal()" is synchronized is a complete mystery as
|all it does is call "loadClass()" which can be synchronized if it needs to
|be.
|
|David
|
|
|> -Original Message-
|> From: marc fleury [mailto:[EMAIL PROTECTED]]
|> Sent: Tuesday, April 23, 2002 8:47 AM
|> To: Bordet, Simone; Jboss-Development@Lists. Sourceforge. Net
|> Subject: RE: [JBoss-dev] Workaround for CL stuff
|>
|>
|> Simone,
|>
|> I still don't see the problem clearly, please help me.
|>
|> |> Where does the "loadClassInternal()" intervene?
|> |
|> |It is called by the JVM when it has to resolve class dependencies.
|>
|> ok
|>
|> |Say you have class A that has a data member of class B.
|> |When a CL is asked to load A, it inspects the class and see that
|> |it needs to load also B. This is done at native level, and while a
|> |classloader is loading class A, the JVM calls loadClassInternal()
|> |on the *same* thread and on the *same* classloader to load class
|> |B; this may trigger another call and so on recursively. If the
|>
|> ok
|>
|> |classloader delegates to some other classloader you can have
|> deadlock.
|>
|> this I don't see, you seem to jump to the conclusion, can you
|> please walk
|> me?
|>
|> |There is a call for loadClass, some more call on java code, then a
|> |call to native method, and if the native method decides it has no
|> |sufficient information it calls back java code, it's java ->
|> |native -> java.
|> |
|> |loadClass()
|> | ...
|> |defineClass() [is this the native one ?]
|> | loadClassInternal()
|> |loadClass()
|>
|> So if loadClassInternal fails it goes to the "loadClass()" in
|> our stack?
|>
|> cl1.loadClass("A")
|> A refers to B
|> cl1.loadClassInternal("B") fails
|> cl1.loadClass("B") is then called.
|>
|> is that correct?
|>
|> And then ...?
|>
|> |A typical scenario for me was this one: jar1 with a set of related
|> |classes, and jar2 with another set, but the 2 jars interacts and
|> |are loaded by different classloaders. When something is "started"
|> |in jar1, not all classes in the jar are loaded; the "start"
|> |triggers loading from jar2 that happens to load some class from
|> |jar1 that was not already loaded.
|>
|> I am lost.
|>
|> Let's say
|> jar1:set1:cl1 has classes A and C
|>
|> jar2:set2:cl2 has class B
|>
|> A and B reference C
|>
|> A starts up loads A, loads class internal C (I assume it
|> finds it) but it is
|> not in the repository (internal call).
|>
|> In a mono threaded JBoss, B is loaded up by cl2.load("B"),
|> which doesn't
|> find it, starts up, needs to load C and
RE: [JBoss-dev] Workaround for CL stuff
thanks this is useful marcf |-Original Message- |From: Dave Smith [mailto:[EMAIL PROTECTED]] |Sent: Monday, April 22, 2002 12:49 PM |To: marc fleury |Cc: lsanders; Jboss-Development@Lists. Sourceforge. Net |Subject: RE: [JBoss-dev] Workaround for CL stuff | | |I brought this thing back to life and passed the URL of the orginal sun |bug that they rejected. Here is a quick stack trace of the deadlock. |Note that you should start jboss with the -Xdebug options so it shows |you what objects that it is trying to lock. See below for my orginal |post | | |"CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting for monitor entry |[0xbb7fe000..0xbb7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43a2c508> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) |at |com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeStream.java) |at |com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7EncodeStream.java) |at com.candata.gateway.Encryption.Sign(Unknown Source) |- locked <43f0c6a0> (a com.candata.gateway.Encryption) |at com.candata.gateway.Encryption.createMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.postMsg(Unknown Source) |at com.candata.gateway.CCRAAbstract.recvMsg(Unknown Source) |at com.candata.gateway.CCRAPoll.run(Unknown Source) |at java.lang.Thread.run(Thread.java:484) | | |"Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting for monitor entry |[0xbe7fe000..0xbe7ffad8] |at java.lang.ClassLoader.loadClass(ClassLoader.java:288) |- waiting to lock <43eefa48> (a |org.jboss.mx.loading.UnifiedClassLoader) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedCla ssLoader.java:180) |at |org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoade rRepository.java:178) |at |org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoade |r.java:217) |at java.lang.ClassLoader.loadClass(ClassLoader.java:255) |at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) |- locked <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) |at java.lang.Class.forName0(Native Method) |at java.lang.Class.forName(Class.java:195) |at |javax.security.auth.login.LoginContext.invoke(LoginContext.java:626) |at |javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) |at |javax.security.auth.login.LoginContext$4.run(LoginContext.java:599) |at java.security.AccessController.doPrivileged(Native Method) |at |javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:596) |at |javax.security.auth.login.LoginContext.login(LoginContext.java:523) |at com.candata.util.beans.CandataClientLogin.login(Unknown |Source) |at com.candata.bbxinterface.BbxJavaInvoker.login(Unknown Source) |at com.candata.bbxinterface.BbxWrapper.run(Unknown Source) |at java.lang.Thread.run(Thread.java:484) | | | |On Mon, 2002-04-22 at 14:49, marc fleury wrote: |> |The players: |> |UnifiedClassLoader A (can load directly class foo) |> |UnifiedClassLoader B (can load directly class bar) |> | |> |Thread 1 (context loader is A): |> | Load new class bar: |> |- synchronize on UCL A |> |- Search UCL B |> |- Attempt to synchronize on UCL B |> | |> |Thread 2 (contect loader is B): |> | Load new class foo: |> |- synchronize on UCL B |> |- Search UCL A |> |- Attempt to synchronize on UCL A |> | |> |Can someone verify if this is accurate? |> |> If this is the case this is simple to solve, just synchronize on the |> repository and thus enforce a single thread mode. Trivial, dead problem. |> Unfortunately I don't think that was the scenario that led to |deadlocks but |> I dont' remember as Sacha and Jung were the ones that got it... |but if that |> is the case that is fairly simple to solve... I can't remember and would |> rather not stay in the uncertain land. Where does the |"loadClassInternal()" |> intervene? |> |> marcf |> |> |> |> ___ |> Jboss-development mailing list |> [EMAIL PROTECTED] |> https://lists.sourcefo
RE: [JBoss-dev] Workaround for CL stuff
Ok, I'm not the expert on this stuff either, but in an effort to help out
here is my explanation of what I think is happening.
The major problem (and the starting point of the whole thing) is that when
classes are being loaded the JVM can call "loadClassInternal()" on multiple
class loaders at the same time. This method is synchronized and so
immediately two different threads have locked two different class loaders,
let us say Thread A has locked ClassLoader A and Thread B has locked
ClassLoader B.
Now loadClassInternal simply calls the loadClass method of the respective
class loader and so if you have a class loader (like the JBoss
UnifiedClassLoader) which can delegate to other class loaders (via
UnifiedLoaderRepository) to load classes you can get a situation where
Thread A now wants to access synchronized methods in ClassLoader B and
Thread B wants to access synchronized methods in ClassLoader A and neither
can because of the earlier locks already gained, hence deadlock.
This bought about by the fact that the private "loadClassInternal()" method
is synchronized and can be called pretty much at any time by the JVM.
The only way you could work around the problem is if you could synchronize
on some common object before loadClassInternal is called, but since this is
called rather unpredictably directly from the JVM I don't know if there is
any way to do this.
Why the F*K "loadClassInternal()" is synchronized is a complete mystery as
all it does is call "loadClass()" which can be synchronized if it needs to
be.
David
> -Original Message-
> From: marc fleury [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, April 23, 2002 8:47 AM
> To: Bordet, Simone; Jboss-Development@Lists. Sourceforge. Net
> Subject: RE: [JBoss-dev] Workaround for CL stuff
>
>
> Simone,
>
> I still don't see the problem clearly, please help me.
>
> |> Where does the "loadClassInternal()" intervene?
> |
> |It is called by the JVM when it has to resolve class dependencies.
>
> ok
>
> |Say you have class A that has a data member of class B.
> |When a CL is asked to load A, it inspects the class and see that
> |it needs to load also B. This is done at native level, and while a
> |classloader is loading class A, the JVM calls loadClassInternal()
> |on the *same* thread and on the *same* classloader to load class
> |B; this may trigger another call and so on recursively. If the
>
> ok
>
> |classloader delegates to some other classloader you can have
> deadlock.
>
> this I don't see, you seem to jump to the conclusion, can you
> please walk
> me?
>
> |There is a call for loadClass, some more call on java code, then a
> |call to native method, and if the native method decides it has no
> |sufficient information it calls back java code, it's java ->
> |native -> java.
> |
> |loadClass()
> | ...
> |defineClass() [is this the native one ?]
> | loadClassInternal()
> |loadClass()
>
> So if loadClassInternal fails it goes to the "loadClass()" in
> our stack?
>
> cl1.loadClass("A")
> A refers to B
> cl1.loadClassInternal("B") fails
> cl1.loadClass("B") is then called.
>
> is that correct?
>
> And then ...?
>
> |A typical scenario for me was this one: jar1 with a set of related
> |classes, and jar2 with another set, but the 2 jars interacts and
> |are loaded by different classloaders. When something is "started"
> |in jar1, not all classes in the jar are loaded; the "start"
> |triggers loading from jar2 that happens to load some class from
> |jar1 that was not already loaded.
>
> I am lost.
>
> Let's say
> jar1:set1:cl1 has classes A and C
>
> jar2:set2:cl2 has class B
>
> A and B reference C
>
> A starts up loads A, loads class internal C (I assume it
> finds it) but it is
> not in the repository (internal call).
>
> In a mono threaded JBoss, B is loaded up by cl2.load("B"),
> which doesn't
> find it, starts up, needs to load C and so cl2.loadCint("B")
> fails goes to
> cl2.load(B) which goes repository.load(B) which then goes to
> cl1.load(B)
> which should work.
>
> Let's imagine a multithreaded JBoss where the service A
> spawns a thread.
>
> please help
>
> marcf
>
>
> ___
> Jboss-development mailing list
> [EMAIL PROTECTED]
> https://lists.sourceforge.net/lists/listinfo/jboss-development
>
> ---
> Incoming mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.350 / Virus Database: 196 - Release Date: 4/17/2002
>
>
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.350 / Virus Database: 196 - Release Date: 4/17/2002
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Simone,
I still don't see the problem clearly, please help me.
|> Where does the "loadClassInternal()" intervene?
|
|It is called by the JVM when it has to resolve class dependencies.
ok
|Say you have class A that has a data member of class B.
|When a CL is asked to load A, it inspects the class and see that
|it needs to load also B. This is done at native level, and while a
|classloader is loading class A, the JVM calls loadClassInternal()
|on the *same* thread and on the *same* classloader to load class
|B; this may trigger another call and so on recursively. If the
ok
|classloader delegates to some other classloader you can have deadlock.
this I don't see, you seem to jump to the conclusion, can you please walk
me?
|There is a call for loadClass, some more call on java code, then a
|call to native method, and if the native method decides it has no
|sufficient information it calls back java code, it's java ->
|native -> java.
|
|loadClass()
| ...
|defineClass() [is this the native one ?]
| loadClassInternal()
|loadClass()
So if loadClassInternal fails it goes to the "loadClass()" in our stack?
cl1.loadClass("A")
A refers to B
cl1.loadClassInternal("B") fails
cl1.loadClass("B") is then called.
is that correct?
And then ...?
|A typical scenario for me was this one: jar1 with a set of related
|classes, and jar2 with another set, but the 2 jars interacts and
|are loaded by different classloaders. When something is "started"
|in jar1, not all classes in the jar are loaded; the "start"
|triggers loading from jar2 that happens to load some class from
|jar1 that was not already loaded.
I am lost.
Let's say
jar1:set1:cl1 has classes A and C
jar2:set2:cl2 has class B
A and B reference C
A starts up loads A, loads class internal C (I assume it finds it) but it is
not in the repository (internal call).
In a mono threaded JBoss, B is loaded up by cl2.load("B"), which doesn't
find it, starts up, needs to load C and so cl2.loadCint("B") fails goes to
cl2.load(B) which goes repository.load(B) which then goes to cl1.load(B)
which should work.
Let's imagine a multithreaded JBoss where the service A spawns a thread.
please help
marcf
___
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Simone, you are a king and we should really find a place for you and your leg space :) thanks I will pause on the training material (almost done) and will look at this marcf |-Original Message- |From: [EMAIL PROTECTED] |[mailto:[EMAIL PROTECTED]]On Behalf Of |Bordet, Simone |Sent: Monday, April 22, 2002 12:27 PM |To: Jboss-Development@Lists. Sourceforge. Net |Subject: RE: [JBoss-dev] Workaround for CL stuff | | |Hi, | |> Where does the "loadClassInternal()" intervene? | |It is called by the JVM when it has to resolve class dependencies. | |Say you have class A that has a data member of class B. |When a CL is asked to load A, it inspects the class and see that |it needs to load also B. This is done at native level, and while a |classloader is loading class A, the JVM calls loadClassInternal() |on the *same* thread and on the *same* classloader to load class |B; this may trigger another call and so on recursively. If the |classloader delegates to some other classloader you can have deadlock. |There is a call for loadClass, some more call on java code, then a |call to native method, and if the native method decides it has no |sufficient information it calls back java code, it's java -> |native -> java. | |loadClass() | ... |defineClass() [is this the native one ?] | loadClassInternal() |loadClass() | |I experienced the deadlock problem in another project at work, and |it happened to me at startup (when not all classes are already |loaded and thus loadClassInternal() is called frequently). | |A typical scenario for me was this one: jar1 with a set of related |classes, and jar2 with another set, but the 2 jars interacts and |are loaded by different classloaders. When something is "started" |in jar1, not all classes in the jar are loaded; the "start" |triggers loading from jar2 that happens to load some class from |jar1 that was not already loaded. | |I solved this problem in my project using reflection in the small |part that had this problem (no data member directly referencing |the class). I don't know if this is doable also in JBoss (waaay |larger than my project :) | |Hope helped | |Simon | |___ |Jboss-development mailing list |[EMAIL PROTECTED] |https://lists.sourceforge.net/lists/listinfo/jboss-development ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
I brought this thing back to life and passed the URL of the orginal sun bug that they rejected. Here is a quick stack trace of the deadlock. Note that you should start jboss with the -Xdebug options so it shows you what objects that it is trying to lock. See below for my orginal post "CCRAPoll" prio=5 tid=0x8184f58 nid=0x64ec waiting for monitor entry [0xbb7fe000..0xbb7ffad8] at java.lang.ClassLoader.loadClass(ClassLoader.java:288) - waiting to lock <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) at org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedClassLoader.java:180) at org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoaderRepository.java:178) at org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoader.java:217) at java.lang.ClassLoader.loadClass(ClassLoader.java:255) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) - locked <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) at com.entrust.toolkit.PKCS7EncodeStream.e(PKCS7EncodeStream.java) at com.entrust.toolkit.PKCS7EncodeStream.f(PKCS7EncodeStream.java) at com.entrust.toolkit.PKCS7EncodeStream.write(PKCS7EncodeStream.java) at com.candata.gateway.Encryption.Sign(Unknown Source) - locked <43f0c6a0> (a com.candata.gateway.Encryption) at com.candata.gateway.Encryption.createMsg(Unknown Source) at com.candata.gateway.CCRAAbstract.postMsg(Unknown Source) at com.candata.gateway.CCRAAbstract.recvMsg(Unknown Source) at com.candata.gateway.CCRAPoll.run(Unknown Source) at java.lang.Thread.run(Thread.java:484) "Thread-20" prio=5 tid=0x81821a8 nid=0x64f9 waiting for monitor entry [0xbe7fe000..0xbe7ffad8] at java.lang.ClassLoader.loadClass(ClassLoader.java:288) - waiting to lock <43eefa48> (a org.jboss.mx.loading.UnifiedClassLoader) at org.jboss.mx.loading.UnifiedClassLoader.loadClassLocally(UnifiedClassLoader.java:180) at org.jboss.mx.loading.UnifiedLoaderRepository.loadClass(UnifiedLoaderRepository.java:178) at org.jboss.mx.loading.UnifiedClassLoader.loadClass(UnifiedClassLoader.java:217) at java.lang.ClassLoader.loadClass(ClassLoader.java:255) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:315) - locked <43a2c508> (a org.jboss.mx.loading.UnifiedClassLoader) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:195) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:626) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:129) at javax.security.auth.login.LoginContext$4.run(LoginContext.java:599) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:596) at javax.security.auth.login.LoginContext.login(LoginContext.java:523) at com.candata.util.beans.CandataClientLogin.login(Unknown Source) at com.candata.bbxinterface.BbxJavaInvoker.login(Unknown Source) at com.candata.bbxinterface.BbxWrapper.run(Unknown Source) at java.lang.Thread.run(Thread.java:484) On Mon, 2002-04-22 at 14:49, marc fleury wrote: > |The players: > |UnifiedClassLoader A (can load directly class foo) > |UnifiedClassLoader B (can load directly class bar) > | > |Thread 1 (context loader is A): > | Load new class bar: > |- synchronize on UCL A > |- Search UCL B > |- Attempt to synchronize on UCL B > | > |Thread 2 (contect loader is B): > | Load new class foo: > |- synchronize on UCL B > |- Search UCL A > |- Attempt to synchronize on UCL A > | > |Can someone verify if this is accurate? > > If this is the case this is simple to solve, just synchronize on the > repository and thus enforce a single thread mode. Trivial, dead problem. > Unfortunately I don't think that was the scenario that led to deadlocks but > I dont' remember as Sacha and Jung were the ones that got it... but if that > is the case that is fairly simple to solve... I can't remember and would > rather not stay in the uncertain land. Where does the "loadClassInternal()" > intervene? > > marcf > > > > ___ > Jboss-development mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/jboss-development ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
Hi, > Where does the "loadClassInternal()" intervene? It is called by the JVM when it has to resolve class dependencies. Say you have class A that has a data member of class B. When a CL is asked to load A, it inspects the class and see that it needs to load also B. This is done at native level, and while a classloader is loading class A, the JVM calls loadClassInternal() on the *same* thread and on the *same* classloader to load class B; this may trigger another call and so on recursively. If the classloader delegates to some other classloader you can have deadlock. There is a call for loadClass, some more call on java code, then a call to native method, and if the native method decides it has no sufficient information it calls back java code, it's java -> native -> java. loadClass() ... defineClass() [is this the native one ?] loadClassInternal() loadClass() I experienced the deadlock problem in another project at work, and it happened to me at startup (when not all classes are already loaded and thus loadClassInternal() is called frequently). A typical scenario for me was this one: jar1 with a set of related classes, and jar2 with another set, but the 2 jars interacts and are loaded by different classloaders. When something is "started" in jar1, not all classes in the jar are loaded; the "start" triggers loading from jar2 that happens to load some class from jar1 that was not already loaded. I solved this problem in my project using reflection in the small part that had this problem (no data member directly referencing the class). I don't know if this is doable also in JBoss (waaay larger than my project :) Hope helped Simon ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
RE: [JBoss-dev] Workaround for CL stuff
|The players: |UnifiedClassLoader A (can load directly class foo) |UnifiedClassLoader B (can load directly class bar) | |Thread 1 (context loader is A): | Load new class bar: |- synchronize on UCL A |- Search UCL B |- Attempt to synchronize on UCL B | |Thread 2 (contect loader is B): | Load new class foo: |- synchronize on UCL B |- Search UCL A |- Attempt to synchronize on UCL A | |Can someone verify if this is accurate? If this is the case this is simple to solve, just synchronize on the repository and thus enforce a single thread mode. Trivial, dead problem. Unfortunately I don't think that was the scenario that led to deadlocks but I dont' remember as Sacha and Jung were the ones that got it... but if that is the case that is fairly simple to solve... I can't remember and would rather not stay in the uncertain land. Where does the "loadClassInternal()" intervene? marcf ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Workaround for CL stuff
Though I've never experienced this, I think this is the problem scenario: The players: UnifiedClassLoader A (can load directly class foo) UnifiedClassLoader B (can load directly class bar) Thread 1 (context loader is A): Load new class bar: - synchronize on UCL A - Search UCL B - Attempt to synchronize on UCL B Thread 2 (contect loader is B): Load new class foo: - synchronize on UCL B - Search UCL A - Attempt to synchronize on UCL A Can someone verify if this is accurate? -Larry > Ok, > > I am sure we can find something. > > I would appreciate a brief description of a CL deadlock scenario due to the > final loadClassInternal. Jung? Sacha? > > It's only software, software is dumb > > marcf > > x > Marc Fleury, Ph.D > President and CEO > JBoss Group, LLC > x > > > > ___ > Jboss-development mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/jboss-development > ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
Re: [JBoss-dev] Workaround for CL stuff
I think one example is when you try to shut down jboss after running the testsuite on jdk 1.4. At least this morning I got stuck and I'm pretty sure there was a cl deadlock. It is not clear to me why shutdown would be deadlocking here however. And this is not simple;-) david jencks On 2002.04.22 14:03:54 -0400 marc fleury wrote: > Ok, > > I am sure we can find something. > > I would appreciate a brief description of a CL deadlock scenario due to > the > final loadClassInternal. Jung? Sacha? > > It's only software, software is dumb > > marcf > > x > Marc Fleury, Ph.D > President and CEO > JBoss Group, LLC > x > > > > ___ > Jboss-development mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/jboss-development > > ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
[JBoss-dev] Workaround for CL stuff
Ok, I am sure we can find something. I would appreciate a brief description of a CL deadlock scenario due to the final loadClassInternal. Jung? Sacha? It's only software, software is dumb marcf x Marc Fleury, Ph.D President and CEO JBoss Group, LLC x ___ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
