[
https://issues.apache.org/jira/browse/CAMEL-17712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Hoffmann reopened CAMEL-17712:
-------------------------------------
Estimated Complexity: Moderate (was: Unknown)
Hello,
I tried the new version 3.14.2 but the memory leak is not fully fixed
unfortunately.
Therefore I digged deeper into the problem and figured out the problem which
lies a bit deeper than I tought.
The problem is when using ThreadLocal variables within CAMEL in multithreaded
environments like application servers. The last fix only works in single
threaded environments unfortunately.
*Background:*
Each thread in Java has a member variable or map "threadlocals" which stores
the values which are managed by the Helper-Class "ThreadLocal", see:
[https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/lang/Thread.java]
The Java-Class ThreadLocal therefore doesn't store any variable by itself but
attaches every value to the current thread (within their threadlocals map).
*Reason for the memory leak:*
I explain the problem with the example of DefaultCamelContext:
The ThreadLocal variable was introduced there in version 3.9:
[https://github.com/apache/camel/blob/camel-3.9.x/core/camel-core-engine/src/main/java/org/apache/camel/impl/DefaultCamelContext.java]
When a multithreaded application server (e.g. Tomcat or glassfish) starts a
CamelContext, the DefaultCamelContext.java stores via the ThreadLocal-class and
member variable OPTIONS the values (OptionHolder) into the current worker
thread. This worker thread might be e.g. the thread worker-1. Some http calls
to the webserver might be processed with the thread worker-2. This thread gets
it's own option values when working with the started CamelContext. This new
options are again attached to the current thread, in this case attached to
worker-2.
If the application is undeployed, this might be processed by worker-3 but
worker-3 can't clean up the options of the DefaultCamelContext which are stored
in other worker-threads. Therefore the worker-1 and worker-2 thread still hold
references to the CAMEL objects and thus they won't get garbage collected.
*Further occurences:*
The same problem occurs in other classes, like DefaultReactiveExecutor.java and
AbstractCamelContext.java.
When using ThreadLocal-classes, the worker-threads of the application server
get "polluted" with Camel-classes and the instances stay in memory, even when
the application is undeployed because the worker-thread of the applicatoin
server are re-used for subsequent http requests across the hosted applications.
Thus the worker threads hold references to Camel classes and can't be garbage
collected.
*Possible solutions:*
I am not sure, why there is a need to use ThreadLocal class in the mentioned
Camel classes because I see no reason that every (worker) thread should have
their own set of camel options, camel workers etc. Maybe the ThreadLocal
classes where just used to prevent multi-threaded issues?
I currently see two options to solve the problem:
1) Make sure the ThreadLocal values are cleaned up before the method exits (via
remove() method)
2) Replace the ThreadLocal variable with something else, if a per-thread
variable is not needed.
I hope I could explain the complex problem well enough. If further information
is needed, just drop a line.
Thank you in advance! Thomas
> Memory leak in DefaultCamelContext reported by Tomcat 10
> --------------------------------------------------------
>
> Key: CAMEL-17712
> URL: https://issues.apache.org/jira/browse/CAMEL-17712
> Project: Camel
> Issue Type: Bug
> Components: camel-core-engine
> Affects Versions: 3.14.1
> Environment: Apache Camel 3.14.1
> Tomcat 10
> Java 17
> Reporter: Thomas Hoffmann
> Assignee: Claus Ibsen
> Priority: Minor
> Fix For: 3.11.6, 3.14.2, 3.16.0
>
>
> Hello,
> we are using Camel inside a Tomcat application. The application also supports
> reloading of the context.
> Unfortunately, we are getting a memory leak report, e.g.
> {code:java}
> [Catalina-utility-1]
> org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks
> The web application [ROOT] created a ThreadLocal with key of type
> [java.lang.ThreadLocal.SuppliedThreadLocal] (value
> [java.lang.ThreadLocal$SuppliedThreadLocal@588bee00]) and a value of type
> [org.apache.camel.impl.DefaultCamelContext.OptionHolder] (value
> [org.apache.camel.impl.DefaultCamelContext$OptionHolder@1d5c4495]) but failed
> to remove it when the web application was stopped. Threads are going to be
> renewed over time to try and avoid a probable memory leak.{code}
>
> Looking at the code, the problem is within the java class DefaultCamelContext
> [https://github.com/apache/camel/blob/main/core/camel-core-engine/src/main/java/org/apache/camel/impl/DefaultCamelContext.java]
> It contains a class variable which is always instantiated with:
> ThreadLocal<OptionHolder> OPTIONS =
> ThreadLocal.withInitial(OptionHolder::new);
> This ThreadLocal is never cleaned up by OPTIONS.remove(), This should be
> added to the shutdown() or stop() handler maybe(?)
> Some additional null checks would have to be implemented as well to make it
> safe.
>
> Thanks,
> Thomas
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
