Hi Thomas.
Got it.
I think it would be best if you opened a Bugzilla entry for this.
One view is that the JRE should not be doing this - or at least doing it
in a way that doesn't run the risk of memory leaks in a Java EE /
Jakarta EE environment. We have raised JRE bugs for issues like this in
the past and they have been fixed.
The counter view is that this thread is not a one-off and is not
short-lived (the typical cases for JRE bugs that have been fixed) and
Tomcat has already acknowledged - with the useContextClassLoader flag -
that it needs to be taking action to ensure that any threads are created
with the expected class loader.
If the consensus is that Tomcat needs to handle this then we'd need to
ensure that every version of authenticate() also set the class loader if
required. We'd probably want to refactor the existing handling to make
sure we don't try setting the class loader twice.
Whatever changes we make we need to be able to reference them back to
the discussion of why we made them and a Bugzilla issue is how we do that.
Thanks,
Mark
On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:
Hello Mark,
thanks for your reply.
I already tried that option. But this flag only controls, how the
InitialDirContext is created.
The worker thread within com.sun.jndi.ldap.Connection is created with that
corresponding classloader the first time.
The problem is when the worker-thread within the com.sun.jndi.ldap.Connection
dies and is re-established again.
In that case, the flag useContextClassLoader is not considered any more because
the InitialDirContext is already instantiated.
The call stack when the InitialDirContext is already instantiated but the
connection gets re-established looks like:
JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch -->
LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.<init>
In this call chain, the flag useContextClassLoader is not used any more as the
InitialDirContext already exists.
The get() method just provides the existing JNDIConnection without switching
any classloader.
Now however, the context classloader of the application is used, independent of the
setting "useContextClassLoader".
Therefore, the second time when the worker thread gets created, it inherits the
classloader of the application and is reported as leaking during undeployment.
Greetings, Thomas
Von: Mark Thomas <ma...@apache.org>
Gesendet: Sonntag, 5. September 2021 11:55
An: users@tomcat.apache.org
Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak
Thomas,
Try setting:
useContextClassLoader="false"
for the JNDIRealm.
Mark
On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
Hello,
we are using the org.apache.catalina.realm.JNDIRealm for authentication of
users against our windows AD.
When undeploying the application, we see the following warning in our logs:
WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to
have started a thread named [Thread-106] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:
java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
The warning is not always shown but quite often.
Summary of the analysis of the problem:
On tomcat startup, the worker-thread is running under the tomcat classloader.
But when a reconnect happens, the thread is running with the classloader of the
web application and gets thus reported.
The details:
Digging into the problem via remote debugging showed the reason how this
happens:
During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is
switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the
attribute "useContextClassLoader".
Afterwards the context-Object is created (createDirContext). Within this
LdapCtx, an LdapClient is used to communicate with the AD-Server.
This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This
connection opens the reported Worker-Thread.
This can be seen at
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java
around line 243 --> worker = Obj.helper.createThread(this);
So far, so good.
Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread
dies. At least, the thread is not visible any more. Maybe because of a timeout
on the AD-server side or something else happened.
If a new user accesses the site, the JNDIRealm is authenticating the user.
This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch -->
LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance
--> Connection.<init>
This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this
time, the thread is connected to the classloader of the web-application.
On undeployment, the thread is thus reported to be orphaned.
It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.
As the authentication is conducted within tomcat, before the application is
triggered, I am not sure if the problem can be tackled on application side.
Thanks in advance,
Thomas
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
