Re: AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
On 06/09/2021 08:47, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thank you for taking a look at it and confirming the behaviour. I will file a bug as suggested. Thanks, I see it. I have added my own comment. I'll give others some time to comment and aim to fix this for the October release round. Mark Thank you and have a good start into the new week! Thomas -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 6. September 2021 09:36 An: users@tomcat.apache.org Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Hi Thomas. Got it. I think it would be best if you opened a Bugzilla entry for this. One view is that the JRE should not be doing this - or at least doing it in a way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE environment. We have raised JRE bugs for issues like this in the past and they have been fixed. The counter view is that this thread is not a one-off and is not short-lived (the typical cases for JRE bugs that have been fixed) and Tomcat has already acknowledged - with the useContextClassLoader flag - that it needs to be taking action to ensure that any threads are created with the expected class loader. If the consensus is that Tomcat needs to handle this then we'd need to ensure that every version of authenticate() also set the class loader if required. We'd probably want to refactor the existing handling to make sure we don't try setting the class loader twice. Whatever changes we make we need to be able to reference them back to the discussion of why we made them and a Bugzilla issue is how we do that. Thanks, Mark On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thanks for your reply. I already tried that option. But this flag only controls, how the InitialDirContext is created. The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time. The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again. In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated. The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like: JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection. In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists. The get() method just provides the existing JNDIConnection without switching any classloader. Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader". Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment. Greetings, Thomas Von: Mark Thomas Gesendet: Sonntag, 5. September 2021 11:55 An: users@tomcat.apache.org Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Thomas, Try setting: useContextClassLoader="false" for the JNDIRealm. Mark On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello, we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD. When undeploying the application, we see the following warning in our logs: WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method) java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr eam.java:115) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja va:168) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja va:140) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket InputRecord.java:448) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet ePacket(SSLSocketInputRecord.java:68) java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord (SSLSocketImpl.java:1104) java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S SLSocketImpl.java:823) java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream .java:252) java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStrea m.java:292) java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream .java:351) java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:8 32) java.base@11.0.
AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
Hello Mark, thank you for taking a look at it and confirming the behaviour. I will file a bug as suggested. Thank you and have a good start into the new week! Thomas -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 6. September 2021 09:36 An: users@tomcat.apache.org Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Hi Thomas. Got it. I think it would be best if you opened a Bugzilla entry for this. One view is that the JRE should not be doing this - or at least doing it in a way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE environment. We have raised JRE bugs for issues like this in the past and they have been fixed. The counter view is that this thread is not a one-off and is not short-lived (the typical cases for JRE bugs that have been fixed) and Tomcat has already acknowledged - with the useContextClassLoader flag - that it needs to be taking action to ensure that any threads are created with the expected class loader. If the consensus is that Tomcat needs to handle this then we'd need to ensure that every version of authenticate() also set the class loader if required. We'd probably want to refactor the existing handling to make sure we don't try setting the class loader twice. Whatever changes we make we need to be able to reference them back to the discussion of why we made them and a Bugzilla issue is how we do that. Thanks, Mark On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote: > Hello Mark, > > thanks for your reply. > I already tried that option. But this flag only controls, how the > InitialDirContext is created. > The worker thread within com.sun.jndi.ldap.Connection is created with that > corresponding classloader the first time. > > The problem is when the worker-thread within the com.sun.jndi.ldap.Connection > dies and is re-established again. > In that case, the flag useContextClassLoader is not considered any more > because the InitialDirContext is already instantiated. > > The call stack when the InitialDirContext is already instantiated but the > connection gets re-established looks like: > JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> > LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> > LdapClient.getInstance --> Connection. > > In this call chain, the flag useContextClassLoader is not used any more as > the InitialDirContext already exists. > The get() method just provides the existing JNDIConnection without switching > any classloader. > Now however, the context classloader of the application is used, independent > of the setting "useContextClassLoader". > Therefore, the second time when the worker thread gets created, it inherits > the classloader of the application and is reported as leaking during > undeployment. > > Greetings, Thomas > > > > > Von: Mark Thomas > Gesendet: Sonntag, 5. September 2021 11:55 > An: users@tomcat.apache.org > Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads > reports memory leak > > Thomas, > > Try setting: > > useContextClassLoader="false" > > for the JNDIRealm. > > Mark > > > On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote: >> Hello, >> >> we are using the org.apache.catalina.realm.JNDIRealm for authentication of >> users against our windows AD. >> When undeploying the application, we see the following warning in our logs: >> >> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp >> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to >> have started a thread named [Thread-106] but has failed to stop it. This is >> very likely to create a memory leak. Stack trace of thread: >> java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native >> Method) >> >> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr >> eam.java:115) >> >> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja >> va:168) >> >> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja >> va:140) >> >> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket >> InputRecord.java:448) >> >> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet >> ePacket(SSLSocketInputRecord.java:68) >> >> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord >> (SSLSocketImpl.java:1104) >> >> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S >> SLSocketImpl.java:823) >> >> java.base@11.0.3/ja
Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
Hi Thomas. Got it. I think it would be best if you opened a Bugzilla entry for this. One view is that the JRE should not be doing this - or at least doing it in a way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE environment. We have raised JRE bugs for issues like this in the past and they have been fixed. The counter view is that this thread is not a one-off and is not short-lived (the typical cases for JRE bugs that have been fixed) and Tomcat has already acknowledged - with the useContextClassLoader flag - that it needs to be taking action to ensure that any threads are created with the expected class loader. If the consensus is that Tomcat needs to handle this then we'd need to ensure that every version of authenticate() also set the class loader if required. We'd probably want to refactor the existing handling to make sure we don't try setting the class loader twice. Whatever changes we make we need to be able to reference them back to the discussion of why we made them and a Bugzilla issue is how we do that. Thanks, Mark On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thanks for your reply. I already tried that option. But this flag only controls, how the InitialDirContext is created. The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time. The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again. In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated. The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like: JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection. In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists. The get() method just provides the existing JNDIConnection without switching any classloader. Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader". Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment. Greetings, Thomas Von: Mark Thomas Gesendet: Sonntag, 5. September 2021 11:55 An: users@tomcat.apache.org Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Thomas, Try setting: useContextClassLoader="false" for the JNDIRealm. Mark On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello, we are using the org.apache.catalina.realm.JNDIRealm for authentication of users against our windows AD. When undeploying the application, we see the following warning in our logs: WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-106] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method) java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168) java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104) java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823) java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351) java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832) java.base@11.0.3/java.lang.Thread.run(Thread.java:834) The warning is not always shown but quite often. Summary of the analysis of the problem: On tomcat startup, the worker-thread is running under the tomcat classloader. But when a reconnect happens, the thread is running with the classloader of the web application and gets thus reported. The details: Digging into the problem via remote debugging showed the reason how this happens: During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the attribute "useContextClassLoader". Afterwards the context-Object is created
AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak
Hello Mark, thanks for your reply. I already tried that option. But this flag only controls, how the InitialDirContext is created. The worker thread within com.sun.jndi.ldap.Connection is created with that corresponding classloader the first time. The problem is when the worker-thread within the com.sun.jndi.ldap.Connection dies and is re-established again. In that case, the flag useContextClassLoader is not considered any more because the InitialDirContext is already instantiated. The call stack when the InitialDirContext is already instantiated but the connection gets re-established looks like: JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection. In this call chain, the flag useContextClassLoader is not used any more as the InitialDirContext already exists. The get() method just provides the existing JNDIConnection without switching any classloader. Now however, the context classloader of the application is used, independent of the setting "useContextClassLoader". Therefore, the second time when the worker thread gets created, it inherits the classloader of the application and is reported as leaking during undeployment. Greetings, Thomas Von: Mark Thomas Gesendet: Sonntag, 5. September 2021 11:55 An: users@tomcat.apache.org Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak Thomas, Try setting: useContextClassLoader="false" for the JNDIRealm. Mark On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote: > Hello, > > we are using the org.apache.catalina.realm.JNDIRealm for authentication of > users against our windows AD. > When undeploying the application, we see the following warning in our logs: > > WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp > ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to > have started a thread named [Thread-106] but has failed to stop it. This is > very likely to create a memory leak. Stack trace of thread: > java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method) > >java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) > java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168) > java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140) > >java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) > >java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) > >java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104) > >java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823) > >java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) > >java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) > >java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351) > java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832) > java.base@11.0.3/java.lang.Thread.run(Thread.java:834) > > The warning is not always shown but quite often. > > Summary of the analysis of the problem: > On tomcat startup, the worker-thread is running under the tomcat classloader. > But when a reconnect happens, the thread is running with the classloader of > the web application and gets thus reported. > > The details: > Digging into the problem via remote debugging showed the reason how this > happens: > During startup, Tomcat is initializing the JNDIRealm. The open-method of > JNDIRealm is switching the classloader between bootstrap-CL and > tomcat-lib-CL, depending on the attribute "useContextClassLoader". > Afterwards the context-Object is created (createDirContext). Within this > LdapCtx, an LdapClient is used to communicate with the AD-Server. > This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. > This connection opens the reported Worker-Thread. > This can be seen at > https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java > around line 243 --> worker = Obj.helper.createThread(this); > > So far, so good. > > Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread > dies. At least, the thread is not visible any more. Maybe because of a > timeout on the AD-server side or something else happened. > If a new user accesses the site, the JNDIRealm is authenticating the user. > This triggers the following chain (path is shortened): > JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen --> > LdapCtx.connect --> LdapClient.getInstance --> Connection. > This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But > this time, the thread is connected to the classloader of the
