subject:"Memory Leak"

Re: Tomcat 9 memory leak message

2024-06-04 Thread Christopher Schultz

Jenny,

On 6/3/24 12:35, Ying Jin wrote:

Chris,

Thanks for your suggestion!

Another question I would like to confirm with you is if we have to remove
the ojdbc jar file from our web application’s web-inf/lib folder or not.

The reason of our concern is that we need to use the same code base to
create a jar file and deploy it to stone branch which needs to have the
ojdbc jar file included in the application.

Therefore, we prefer to include the ojdbc jar file in the application to
avoid confusion, however, we’re not sure if this will cause any issues for
the web application’s deployment or not on tomcat server.

Using WEB-INF/lib for your JDBC driver should work, but your JDBC driver
map have a bug which causes leaks. If you cannot get the driver to
de-register properly, you have two choices:

1. Ignore the warning messages and eventually run out or memory

2. Move (not copy) your driver to CATALINA_BASE/lib

Hope that helps,
-chris

On Sun, Jun 2, 2024 at 1:57 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

Jenny,

(Apologies for top-posting)

“Safely ignored” can mean many things. You are only in danger of running
out of heap space. So if you aren’t worried about that, feel free to ignore
the error message.

If it were my system, I would want to ensure a clean unload of the driver
when the application shuts down.

But I don’t think it will harm anything besides your memory usage.

-chris

On Jun 1, 2024, at 00:18, Ying Jin wrote:

Chris,

Thanks for your reply!

We already removed the ojdbc8.jar file from the application's Web-inf/lib
folder as suggested in the following post, however, we still got the
warning messages below after the application is deployed to the Tomcat 9
server.

https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [InterruptTimer] but has failed to stop it. This

very likely to create a memory leak. Stack trace of thread:

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
failed to stop it. This is very likely to create a memory leak. Stack

trace

of thread:

I also read some posts saying these warning messages can be safely

ignored

if the Tomcat version is greater than 7.0. I'm not sure if this is

correct

or not.

Please advise,

Many thanks!
Jenny

On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

Jenny,

On 5/31/24 14:52, Ying Jin wrote:
We removed the ojdbc8 driver jar from web-inf/lib from the web
application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
we still can see the following memory link warning message whenever we
redeploy the web application. We use the Tomcat 9 server in the Linux
environment.

This list strips attachments. Can you re-post with text-only?

The other warning message is about the "validateFile Problem with jar
file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
these warning messages or not.

It would be great if you can shed some light on this issue.

If the message is something like "driver cannot be unloaded" then check
Tomcat with a debugger or even something like JVisualVM to see how many
WebappClassLoaders you have in memory.

If the driver causes the web application ClassLoader to be "pinned" in
memory, then it will never be removed and all those classes will
continue to use-up heap space until you restart the JVM. This gets worse
every time you reload your application without restarting the JVM. The
Manager application web UI can help you diagnose these a little.

The validation problem with the Jolokia JAR file will depend upon
exactly what it says. I would first get a replacement copy of the
Jolokia JAR file before bothering to try to diagnose it any further.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9 memory leak message

2024-06-03 Thread Ying Jin

Terence, thanks for your reply! best, Jenny
On Sun, Jun 2, 2024 at 3:11 AM Terence M. Bandoian 
wrote:

> A long time ago (Tomcat 7), I would see a Tomcat memory leak detection
> error on shutdown if the JDBC driver was located in WEB-INF/lib but not
> if it was in the Tomcat lib directory. Explicitly de-registering the
> driver with a ServletContextListener on shutdown eliminated the error.
> I'm not sure if that's still necessary or if it applies in your case.
>
> -Terence Bandoian
>
> On 6/2/2024 1:56 AM, Christopher Schultz wrote:
> > Jenny,
> >
> > (Apologies for top-posting)
> >
> > “Safely ignored” can mean many things. You are only in danger of running
> out of heap space. So if you aren’t worried about that, feel free to ignore
> the error message.
> >
> > If it were my system, I would want to ensure a clean unload of the
> driver when the application shuts down.
> >
> > But I don’t think it will harm anything besides your memory usage.
> >
> > -chris
> >
> >> On Jun 1, 2024, at 00:18, Ying Jin  wrote:
> >>
> >> Chris,
> >>
> >> Thanks for your reply!
> >>
> >> We already removed the ojdbc8.jar file from the application's
> Web-inf/lib
> >> folder as suggested in the following post, however, we still got the
> >> warning messages below after the application is deployed to the Tomcat 9
> >> server.
> >>
> >>
> https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder
> >>
> >> WARNING: The web application [Our Web Application Name] appears to have
> >> started a thread named [InterruptTimer] but has failed to stop it. This
> is
> >> very likely to create a memory leak. Stack trace of thread:
> >>
> >> WARNING: The web application [Our Web Application Name] appears to have
> >> started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but
> has
> >> failed to stop it. This is very likely to create a memory leak. Stack
> trace
> >> of thread:
> >>
> >> I also read some posts saying these warning messages can be safely
> ignored
> >> if the Tomcat version is greater than 7.0. I'm not sure if this is
> correct
> >> or not.
> >>
> >> Please advise,
> >>
> >> Many thanks!
> >> Jenny
> >>
> >>> On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
> >>> ch...@christopherschultz.net> wrote:
> >>>
> >>> Jenny,
> >>>
> >>>> On 5/31/24 14:52, Ying Jin wrote:
> >>>> We removed the ojdbc8 driver jar from web-inf/lib from the web
> >>>> application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
> >>>> we still can see the following memory link warning message whenever we
> >>>> redeploy the web application. We use the Tomcat 9 server in the Linux
> >>>> environment.
> >>> This list strips attachments. Can you re-post with text-only?
> >>>
> >>>> The other warning message is about the "validateFile Problem with jar
> >>>> file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
> >>>> these warning messages or not.
> >>>>
> >>>> It would be great if you can shed some light on this issue.
> >>> If the message is something like "driver cannot be unloaded" then check
> >>> Tomcat with a debugger or even something like JVisualVM to see how many
> >>> WebappClassLoaders you have in memory.
> >>>
> >>> If the driver causes the web application ClassLoader to be "pinned" in
> >>> memory, then it will never be removed and all those classes will
> >>> continue to use-up heap space until you restart the JVM. This gets
> worse
> >>> every time you reload your application without restarting the JVM. The
> >>> Manager application web UI can help you diagnose these a little.
> >>>
> >>> The validation problem with the Jolokia JAR file will depend upon
> >>> exactly what it says. I would first get a replacement copy of the
> >>> Jolokia JAR file before bothering to try to diagnose it any further.
> >>>
> >>> -chris
> >>>
> >>> -
> >>> To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail:users-h...@tomcat.apache.org
> >>>
> >>>
> > -
> > To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail:users-h...@tomcat.apache.org
> >
>
>

Re: Tomcat 9 memory leak message

2024-06-03 Thread Ying Jin

Chris,

Thanks for your suggestion!

Another question I would like to confirm with you is if we have to remove
the ojdbc jar file from our web application’s web-inf/lib folder or not.

The reason of our concern is that we need to use the same code base to
create a jar file and deploy it to stone branch which needs to have the
ojdbc jar file included in the application.

Therefore, we prefer to include the ojdbc jar file in the application to
avoid confusion, however, we’re not sure if this will cause any issues for
the web application’s deployment or not on tomcat server.

Thanks again for your help!
Jenny

On Sun, Jun 2, 2024 at 1:57 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Jenny,
>
> (Apologies for top-posting)
>
> “Safely ignored” can mean many things. You are only in danger of running
> out of heap space. So if you aren’t worried about that, feel free to ignore
> the error message.
>
> If it were my system, I would want to ensure a clean unload of the driver
> when the application shuts down.
>
> But I don’t think it will harm anything besides your memory usage.
>
> -chris
>
> > On Jun 1, 2024, at 00:18, Ying Jin  wrote:
> >
> > Chris,
> >
> > Thanks for your reply!
> >
> > We already removed the ojdbc8.jar file from the application's Web-inf/lib
> > folder as suggested in the following post, however, we still got the
> > warning messages below after the application is deployed to the Tomcat 9
> > server.
> >
> >
> https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder
> >
> > WARNING: The web application [Our Web Application Name] appears to have
> > started a thread named [InterruptTimer] but has failed to stop it. This
> is
> > very likely to create a memory leak. Stack trace of thread:
> >
> > WARNING: The web application [Our Web Application Name] appears to have
> > started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
> > failed to stop it. This is very likely to create a memory leak. Stack
> trace
> > of thread:
> >
> > I also read some posts saying these warning messages can be safely
> ignored
> > if the Tomcat version is greater than 7.0. I'm not sure if this is
> correct
> > or not.
> >
> > Please advise,
> >
> > Many thanks!
> > Jenny
> >
> >> On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
> >> ch...@christopherschultz.net> wrote:
> >>
> >> Jenny,
> >>
> >>> On 5/31/24 14:52, Ying Jin wrote:
> >>> We removed the ojdbc8 driver jar from web-inf/lib from the web
> >>> application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
> >>> we still can see the following memory link warning message whenever we
> >>> redeploy the web application. We use the Tomcat 9 server in the Linux
> >>> environment.
> >>
> >> This list strips attachments. Can you re-post with text-only?
> >>
> >>> The other warning message is about the "validateFile Problem with jar
> >>> file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
> >>> these warning messages or not.
> >>>
> >>> It would be great if you can shed some light on this issue.
> >>
> >> If the message is something like "driver cannot be unloaded" then check
> >> Tomcat with a debugger or even something like JVisualVM to see how many
> >> WebappClassLoaders you have in memory.
> >>
> >> If the driver causes the web application ClassLoader to be "pinned" in
> >> memory, then it will never be removed and all those classes will
> >> continue to use-up heap space until you restart the JVM. This gets worse
> >> every time you reload your application without restarting the JVM. The
> >> Manager application web UI can help you diagnose these a little.
> >>
> >> The validation problem with the Jolokia JAR file will depend upon
> >> exactly what it says. I would first get a replacement copy of the
> >> Jolokia JAR file before bothering to try to diagnose it any further.
> >>
> >> -chris
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat 9 memory leak message

2024-06-02 Thread Terence M. Bandoian

A long time ago (Tomcat 7), I would see a Tomcat memory leak detection
error on shutdown if the JDBC driver was located in WEB-INF/lib but not
if it was in the Tomcat lib directory. Explicitly de-registering the
driver with a ServletContextListener on shutdown eliminated the error.
I'm not sure if that's still necessary or if it applies in your case.

-Terence Bandoian

On 6/2/2024 1:56 AM, Christopher Schultz wrote:

Jenny,

(Apologies for top-posting)

“Safely ignored” can mean many things. You are only in danger of running out of
heap space. So if you aren’t worried about that, feel free to ignore the error
message.

If it were my system, I would want to ensure a clean unload of the driver when
the application shuts down.

But I don’t think it will harm anything besides your memory usage.

-chris

On Jun 1, 2024, at 00:18, Ying Jin wrote:

Chris,

Thanks for your reply!

https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [InterruptTimer] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:

I also read some posts saying these warning messages can be safely ignored
if the Tomcat version is greater than 7.0. I'm not sure if this is correct
or not.

Please advise,

Many thanks!
Jenny

On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

Jenny,

This list strips attachments. Can you re-post with text-only?

The other warning message is about the "validateFile Problem with jar
file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
these warning messages or not.

It would be great if you can shed some light on this issue.

If the message is something like "driver cannot be unloaded" then check
Tomcat with a debugger or even something like JVisualVM to see how many
WebappClassLoaders you have in memory.

-chris

-
To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:users-h...@tomcat.apache.org

Re: Tomcat 9 memory leak message

2024-06-02 Thread Christopher Schultz

Jenny,

(Apologies for top-posting)

“Safely ignored” can mean many things. You are only in danger of running out of 
heap space. So if you aren’t worried about that, feel free to ignore the error 
message.

If it were my system, I would want to ensure a clean unload of the driver when 
the application shuts down.

But I don’t think it will harm anything besides your memory usage.

-chris

> On Jun 1, 2024, at 00:18, Ying Jin  wrote:
> 
> Chris,
> 
> Thanks for your reply!
> 
> We already removed the ojdbc8.jar file from the application's Web-inf/lib
> folder as suggested in the following post, however, we still got the
> warning messages below after the application is deployed to the Tomcat 9
> server.
> 
> https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder
> 
> WARNING: The web application [Our Web Application Name] appears to have
> started a thread named [InterruptTimer] but has failed to stop it. This is
> very likely to create a memory leak. Stack trace of thread:
> 
> WARNING: The web application [Our Web Application Name] appears to have
> started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
> failed to stop it. This is very likely to create a memory leak. Stack trace
> of thread:
> 
> I also read some posts saying these warning messages can be safely ignored
> if the Tomcat version is greater than 7.0. I'm not sure if this is correct
> or not.
> 
> Please advise,
> 
> Many thanks!
> Jenny
> 
>> On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
>> 
>> Jenny,
>> 
>>> On 5/31/24 14:52, Ying Jin wrote:
>>> We removed the ojdbc8 driver jar from web-inf/lib from the web
>>> application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
>>> we still can see the following memory link warning message whenever we
>>> redeploy the web application. We use the Tomcat 9 server in the Linux
>>> environment.
>> 
>> This list strips attachments. Can you re-post with text-only?
>> 
>>> The other warning message is about the "validateFile Problem with jar
>>> file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
>>> these warning messages or not.
>>> 
>>> It would be great if you can shed some light on this issue.
>> 
>> If the message is something like "driver cannot be unloaded" then check
>> Tomcat with a debugger or even something like JVisualVM to see how many
>> WebappClassLoaders you have in memory.
>> 
>> If the driver causes the web application ClassLoader to be "pinned" in
>> memory, then it will never be removed and all those classes will
>> continue to use-up heap space until you restart the JVM. This gets worse
>> every time you reload your application without restarting the JVM. The
>> Manager application web UI can help you diagnose these a little.
>> 
>> The validation problem with the Jolokia JAR file will depend upon
>> exactly what it says. I would first get a replacement copy of the
>> Jolokia JAR file before bothering to try to diagnose it any further.
>> 
>> -chris
>> 
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9 memory leak message

2024-06-01 Thread Ying Jin

Chris,

Thanks so much  for your suggestions!

We use Oracle version 19. I read the post you mentioned and found that this
warning message can be safely ignored. Please correct me if I'm wrong.

Thanks again!
Jenny



On Sat, Jun 1, 2024 at 7:57 AM Thomas Hoffmann (Speed4Trade GmbH)
 wrote:

> Hello Jenny,
>
> > -Ursprüngliche Nachricht-
> > Von: Ying Jin 
> > Gesendet: Samstag, 1. Juni 2024 00:18
> > An: Tomcat Users List 
> > Betreff: Re: Tomcat 9 memory leak message
> >
> > Chris,
> >
> > Thanks for your reply!
> >
> > We already removed the ojdbc8.jar file from the application's Web-inf/lib
> > folder as suggested in the following post, however, we still got the
> warning
> > messages below after the application is deployed to the Tomcat 9 server.
> >
> > https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-
> > put-in-tomcat-home-lib-folder
> >
> > WARNING: The web application [Our Web Application Name] appears to have
> > started a thread named [InterruptTimer] but has failed to stop it. This
> is very
> > likely to create a memory leak. Stack trace of thread:
> >
> > WARNING: The web application [Our Web Application Name] appears to have
> > started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
> > failed to stop it. This is very likely to create a memory leak. Stack
> trace of
> > thread:
>
> I don't know which Oracle Version you are using.
> The JDBC-Driver seems to start some threads, but they are not stopped at
> shutdown of the application.
> One of the issues seems to be resolved in a newer version:
>
> https://stackoverflow.com/questions/78300946/how-to-not-run-or-stop-oracle-23-jdbc-driver-diagnostic-clock-thread
>
> >
> > I also read some posts saying these warning messages can be safely
> ignored if
> > the Tomcat version is greater than 7.0. I'm not sure if this is correct
> or not.
> >
> > Please advise,
> >
> > Many thanks!
> > Jenny
> >
> > On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> > > Jenny,
> > >
> > > On 5/31/24 14:52, Ying Jin wrote:
> > > > We removed the ojdbc8 driver jar from web-inf/lib from the web
> > > > application and kept the ojdbc8 jar file in the Tomcat/lib folder,
> > > > but we still can see the following memory link warning message
> > > > whenever we redeploy the web application. We use the Tomcat 9 server
> > > > in the Linux environment.
> > >
> > > This list strips attachments. Can you re-post with text-only?
> > >
> > > > The other warning message is about the "validateFile Problem with
> > > > jar file /tomcat/lib/jolokia.jar. My question is if we can safely
> > > > ignore these warning messages or not.
> > > >
> > > > It would be great if you can shed some light on this issue.
> > >
> > > If the message is something like "driver cannot be unloaded" then
> > > check Tomcat with a debugger or even something like JVisualVM to see
> > > how many WebappClassLoaders you have in memory.
> > >
> > > If the driver causes the web application ClassLoader to be "pinned" in
> > > memory, then it will never be removed and all those classes will
> > > continue to use-up heap space until you restart the JVM. This gets
> > > worse every time you reload your application without restarting the
> > > JVM. The Manager application web UI can help you diagnose these a
> little.
> > >
> > > The validation problem with the Jolokia JAR file will depend upon
> > > exactly what it says. I would first get a replacement copy of the
> > > Jolokia JAR file before bothering to try to diagnose it any further.
> > >
> > > -chris
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> > >
>

AW: Tomcat 9 memory leak message

2024-06-01 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Jenny,

> -Ursprüngliche Nachricht-
> Von: Ying Jin 
> Gesendet: Samstag, 1. Juni 2024 00:18
> An: Tomcat Users List 
> Betreff: Re: Tomcat 9 memory leak message
> 
> Chris,
> 
> Thanks for your reply!
> 
> We already removed the ojdbc8.jar file from the application's Web-inf/lib
> folder as suggested in the following post, however, we still got the warning
> messages below after the application is deployed to the Tomcat 9 server.
> 
> https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-
> put-in-tomcat-home-lib-folder
> 
> WARNING: The web application [Our Web Application Name] appears to have
> started a thread named [InterruptTimer] but has failed to stop it. This is 
> very
> likely to create a memory leak. Stack trace of thread:
> 
> WARNING: The web application [Our Web Application Name] appears to have
> started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
> failed to stop it. This is very likely to create a memory leak. Stack trace of
> thread:

I don't know which Oracle Version you are using.
The JDBC-Driver seems to start some threads, but they are not stopped at 
shutdown of the application.
One of the issues seems to be resolved in a newer version:
https://stackoverflow.com/questions/78300946/how-to-not-run-or-stop-oracle-23-jdbc-driver-diagnostic-clock-thread

> 
> I also read some posts saying these warning messages can be safely ignored if
> the Tomcat version is greater than 7.0. I'm not sure if this is correct or 
> not.
> 
> Please advise,
> 
> Many thanks!
> Jenny
> 
> On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
> 
> > Jenny,
> >
> > On 5/31/24 14:52, Ying Jin wrote:
> > > We removed the ojdbc8 driver jar from web-inf/lib from the web
> > > application and kept the ojdbc8 jar file in the Tomcat/lib folder,
> > > but we still can see the following memory link warning message
> > > whenever we redeploy the web application. We use the Tomcat 9 server
> > > in the Linux environment.
> >
> > This list strips attachments. Can you re-post with text-only?
> >
> > > The other warning message is about the "validateFile Problem with
> > > jar file /tomcat/lib/jolokia.jar. My question is if we can safely
> > > ignore these warning messages or not.
> > >
> > > It would be great if you can shed some light on this issue.
> >
> > If the message is something like "driver cannot be unloaded" then
> > check Tomcat with a debugger or even something like JVisualVM to see
> > how many WebappClassLoaders you have in memory.
> >
> > If the driver causes the web application ClassLoader to be "pinned" in
> > memory, then it will never be removed and all those classes will
> > continue to use-up heap space until you restart the JVM. This gets
> > worse every time you reload your application without restarting the
> > JVM. The Manager application web UI can help you diagnose these a little.
> >
> > The validation problem with the Jolokia JAR file will depend upon
> > exactly what it says. I would first get a replacement copy of the
> > Jolokia JAR file before bothering to try to diagnose it any further.
> >
> > -chris
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >

Re: Tomcat 9 memory leak message

2024-05-31 Thread Ying Jin

Chris,

Thanks for your reply!

We already removed the ojdbc8.jar file from the application's Web-inf/lib
folder as suggested in the following post, however, we still got the
warning messages below after the application is deployed to the Tomcat 9
server.

https://stackoverflow.com/questions/6981564/why-must-the-jdbc-driver-be-put-in-tomcat-home-lib-folder

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [InterruptTimer] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:

WARNING: The web application [Our Web Application Name] appears to have
started a thread named [oracle.jdbc.diagnostics.Diagnostic.CLOCK] but has
failed to stop it. This is very likely to create a memory leak. Stack trace
of thread:

I also read some posts saying these warning messages can be safely ignored
if the Tomcat version is greater than 7.0. I'm not sure if this is correct
or not.

Please advise,

Many thanks!
Jenny

On Fri, May 31, 2024 at 3:50 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Jenny,
>
> On 5/31/24 14:52, Ying Jin wrote:
> > We removed the ojdbc8 driver jar from web-inf/lib from the web
> > application and kept the ojdbc8 jar file in the Tomcat/lib folder, but
> > we still can see the following memory link warning message whenever we
> > redeploy the web application. We use the Tomcat 9 server in the Linux
> > environment.
>
> This list strips attachments. Can you re-post with text-only?
>
> > The other warning message is about the "validateFile Problem with jar
> > file /tomcat/lib/jolokia.jar. My question is if we can safely ignore
> > these warning messages or not.
> >
> > It would be great if you can shed some light on this issue.
>
> If the message is something like "driver cannot be unloaded" then check
> Tomcat with a debugger or even something like JVisualVM to see how many
> WebappClassLoaders you have in memory.
>
> If the driver causes the web application ClassLoader to be "pinned" in
> memory, then it will never be removed and all those classes will
> continue to use-up heap space until you restart the JVM. This gets worse
> every time you reload your application without restarting the JVM. The
> Manager application web UI can help you diagnose these a little.
>
> The validation problem with the Jolokia JAR file will depend upon
> exactly what it says. I would first get a replacement copy of the
> Jolokia JAR file before bothering to try to diagnose it any further.
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat 9 memory leak message

2024-05-31 Thread Christopher Schultz


Jenny,

On 5/31/24 14:52, Ying Jin wrote:
We removed the ojdbc8 driver jar from web-inf/lib from the web 
application and kept the ojdbc8 jar file in the Tomcat/lib folder, but 
we still can see the following memory link warning message whenever we 
redeploy the web application. We use the Tomcat 9 server in the Linux 
environment.


This list strips attachments. Can you re-post with text-only?

The other warning message is about the "validateFile Problem with jar 
file /tomcat/lib/jolokia.jar. My question is if we can safely ignore 
these warning messages or not.


It would be great if you can shed some light on this issue.


If the message is something like "driver cannot be unloaded" then check 
Tomcat with a debugger or even something like JVisualVM to see how many 
WebappClassLoaders you have in memory.


If the driver causes the web application ClassLoader to be "pinned" in 
memory, then it will never be removed and all those classes will 
continue to use-up heap space until you restart the JVM. This gets worse 
every time you reload your application without restarting the JVM. The 
Manager application web UI can help you diagnose these a little.


The validation problem with the Jolokia JAR file will depend upon 
exactly what it says. I would first get a replacement copy of the 
Jolokia JAR file before bothering to try to diagnose it any further.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 9 memory leak message

2024-05-31 Thread Ying Jin

We removed the ojdbc8 driver jar from web-inf/lib from the web application
and kept the ojdbc8 jar file in the Tomcat/lib folder, but we still can see
the following memory link warning message whenever we redeploy the web
application. We use the Tomcat 9 server in the Linux environment.

The other warning message is about the "validateFile Problem with jar file
/tomcat/lib/jolokia.jar. My question is if we can safely ignore these
warning messages or not.

It would be great if you can shed some light on this issue.
[image: image.png]


Your help is much appreciated!
Jenny

Re: Memory leak in EncodingDetector?

2024-03-22 Thread Simon Niederberger

Very cool Chris, thanks for the quick reaction!

Simon out


On Fri, Mar 22, 2024 at 1:41 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Simon,
>
> On 3/21/24 12:39, Simon Niederberger wrote:
> > Hi Chris
> >
> > Personally I'd go with
> >
> > XML_INPUT_FACTORY =
> > XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
> > EncodingDetector.class.getClassLoader());
> >
> > allowing me to place my own JAR in common/lib if I really want to (the
> > only scenario I can think of is an edge-case where there's a bug in
> > the JVM XMLInputFactory and no upgrade path is available, so I'd place
> > my own JAR in common/lib).
> >
> >> I wonder if there really are any use-cases for applications wanting
> Tomcat to specify the XMLInputFactory *to be used for JSP*
> >
> > That sounds pretty far-fetched. Our use case is simply that we deploy
> > frequently (several times a day) on a Tomcat 10.1 installation, and
> > have various reasons for not going down the Docker/container/FAT JAR
> > path. So, making sure we don't have memory leaks is relevant.
>
> Done and done.
>
> Feel free to put the fix in yourself, or grab 10.1.x from GitHub, or wit
> for Tomcat 10.1.21 to be released (10.1.20 has already been rolled and
> the voting will end shortly, so you'll have to wait for the next release).
>
> -chris
>
> > On Thu, Mar 21, 2024 at 1:44 PM Christopher Schultz
> >  wrote:
> >>
> >> Simon,
> >>
> >> On 3/20/24 15:36, Simon Niederberger wrote:
> >>>> What if you create an empty jaxp.properties file and make it
> available to the common ClassLoader (e.g. in
> lib/empty-jaxp.jar:/jaxp.properties) -- does that prevent the problem?
> >>   >
> >>> I think that boils down to what I'm already doing with the system
> >>> property, getting to the FactoryFinder before it uses
> >>> findServiceProvider(). Haven't tried it, but I'm sure it would work.
> >>
> >> The difference is that Tomcat can ship it as a part of the distribution,
> >> while we shouldn't really be setting system properties like that. It
> >> might prevent, for example, applications using their own preferred
> >> implementation.
> >>
> >>> imho, Tomcat's EncodingDetector should init with either
> >>>
> >>> XML_INPUT_FACTORY = XMLInputFactory.newDefaultFactory();
> >>>
> >>> to just always use the JVM XMLInputFactory, or then
> >>>
> >>> XML_INPUT_FACTORY =
> >>> XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
> >>> EncodingDetector.class.getClassLoader());
> >>>
> >>> to honor it's own classloader (maybe
> >>> EncodingDetector.class.getClassLoader() is the wrong approach,
> >>> basically something getTomcatCommonClassloader())
> >>
> >> Yeah, that's probably better than an empty properties file wrapped in a
> >> JAR file wrapped in an enigma.
> >>
> >> We also have the option of using the JreMemoryLeakPreventionListener for
> >> this. There are already some XML-related protections in there, though
> >> this one is not specifically there.
> >>
> >> Do you have a preferred technique for fixing this? All of those
> >> suggestions seem equally reasonable. I think I have a slight preference
> >> for passing the JSP compiler's ClassLoader in to the factory method:
> >>
> >> XML_INPUT_FACTORY =
> >>   XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
> >>   EncodingDetector.class.getClassLoader());
> >>
> >> If you absolutey need to get your JSP compiler to have a custom
> >> XMLInputFactory, you can copy the JSP compiler into your application and
> >> it will use that ClassLoader instead.
> >>
> >>> What I just don't get is why there's so little online about others
> >>> havingEncodingDetector similar issues. Spring + libs like CXF or
> >>> jackson-dataformat-xml are common, both those libs have transitive
> >>> dependencies on woodstox-core. Throw in ehcache, also common, and your
> >>> webapp won't undeploy if it's the first webapp to load a JSP and thus
> >>> clinit EncodingDetector. Maybe the public has just given up on clean
> >>> undeploying.
> >>
> >> My guess is that most people don't really care too much about clean
> >> shutdowns of their applications. They either bring their applications up
> >> and down along with the whole JVM (

Re: Memory leak in EncodingDetector?

2024-03-22 Thread Christopher Schultz


Simon,

On 3/21/24 12:39, Simon Niederberger wrote:

Hi Chris

Personally I'd go with

XML_INPUT_FACTORY =
XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
EncodingDetector.class.getClassLoader());

allowing me to place my own JAR in common/lib if I really want to (the
only scenario I can think of is an edge-case where there's a bug in
the JVM XMLInputFactory and no upgrade path is available, so I'd place
my own JAR in common/lib).


I wonder if there really are any use-cases for applications wanting Tomcat to 
specify the XMLInputFactory *to be used for JSP*


That sounds pretty far-fetched. Our use case is simply that we deploy
frequently (several times a day) on a Tomcat 10.1 installation, and
have various reasons for not going down the Docker/container/FAT JAR
path. So, making sure we don't have memory leaks is relevant.


Done and done.

Feel free to put the fix in yourself, or grab 10.1.x from GitHub, or wit 
for Tomcat 10.1.21 to be released (10.1.20 has already been rolled and 
the voting will end shortly, so you'll have to wait for the next release).


-chris


On Thu, Mar 21, 2024 at 1:44 PM Christopher Schultz
 wrote:


Simon,

On 3/20/24 15:36, Simon Niederberger wrote:

What if you create an empty jaxp.properties file and make it available to the 
common ClassLoader (e.g. in lib/empty-jaxp.jar:/jaxp.properties) -- does that 
prevent the problem?

  >

I think that boils down to what I'm already doing with the system
property, getting to the FactoryFinder before it uses
findServiceProvider(). Haven't tried it, but I'm sure it would work.


The difference is that Tomcat can ship it as a part of the distribution,
while we shouldn't really be setting system properties like that. It
might prevent, for example, applications using their own preferred
implementation.


imho, Tomcat's EncodingDetector should init with either

XML_INPUT_FACTORY = XMLInputFactory.newDefaultFactory();

to just always use the JVM XMLInputFactory, or then

XML_INPUT_FACTORY =
XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
EncodingDetector.class.getClassLoader());

to honor it's own classloader (maybe
EncodingDetector.class.getClassLoader() is the wrong approach,
basically something getTomcatCommonClassloader())


Yeah, that's probably better than an empty properties file wrapped in a
JAR file wrapped in an enigma.

We also have the option of using the JreMemoryLeakPreventionListener for
this. There are already some XML-related protections in there, though
this one is not specifically there.

Do you have a preferred technique for fixing this? All of those
suggestions seem equally reasonable. I think I have a slight preference
for passing the JSP compiler's ClassLoader in to the factory method:

XML_INPUT_FACTORY =
  XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
  EncodingDetector.class.getClassLoader());

If you absolutey need to get your JSP compiler to have a custom
XMLInputFactory, you can copy the JSP compiler into your application and
it will use that ClassLoader instead.


What I just don't get is why there's so little online about others
havingEncodingDetector similar issues. Spring + libs like CXF or
jackson-dataformat-xml are common, both those libs have transitive
dependencies on woodstox-core. Throw in ehcache, also common, and your
webapp won't undeploy if it's the first webapp to load a JSP and thus
clinit EncodingDetector. Maybe the public has just given up on clean
undeploying.


My guess is that most people don't really care too much about clean
shutdowns of their applications. They either bring their applications up
and down along with the whole JVM (either standalone or via Docker,
etc.) or they use an embedded Tomcat where their own application hosts
Tomcat and they start/stop that.

Or they disable/ignore memory-leak detection log messages. *shrug*

I wonder if there really are any use-cases for applications wanting
Tomcat to specify the XMLInputFactory *to be used for JSP*. I suspect not.

-chris


On Wed, Mar 20, 2024 at 7:01 PM Christopher Schultz
 wrote:


Simon,

On 3/20/24 09:59, Simon Niederberger wrote:

The whole thing is caused by Maven dependencies which pull in
com.fasterxml.woodstox:woodstox-core. The WstxInputFactory has a

@ServiceProvider(XMLInputFactory.class)

annotation, where ServiceProvider is
org.ehcache.spi.service.ServiceProvider. I didn't manage to trace the
key code section, but I do find that the
javax.xml.stream.FactoryFinder then ends up finding WstxInputFactory
as registered service provider. As WstxInputFactory is not on the
common classpath (it's in WEB-INF/lib), I assume it's the webapp
classloader which is used. Below is the stacktrace where
EncodingDetector clinit happens (I defined a
ch.want.funnel.FunnelApp$DelegatingXMLInputFactory to get
stacktraces):

Currently I'm setting

System.setProperty("javax.xml.stream.XMLInputFactory",
"com.sun.xml.internal.stream.XMLInputFactoryImpl");
t

Re: Memory leak in EncodingDetector?

2024-03-21 Thread Simon Niederberger

Hi Chris

Personally I'd go with

XML_INPUT_FACTORY =
XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
EncodingDetector.class.getClassLoader());

allowing me to place my own JAR in common/lib if I really want to (the
only scenario I can think of is an edge-case where there's a bug in
the JVM XMLInputFactory and no upgrade path is available, so I'd place
my own JAR in common/lib).

> I wonder if there really are any use-cases for applications wanting Tomcat to 
> specify the XMLInputFactory *to be used for JSP*

That sounds pretty far-fetched. Our use case is simply that we deploy
frequently (several times a day) on a Tomcat 10.1 installation, and
have various reasons for not going down the Docker/container/FAT JAR
path. So, making sure we don't have memory leaks is relevant.

Simon





On Thu, Mar 21, 2024 at 1:44 PM Christopher Schultz
 wrote:
>
> Simon,
>
> On 3/20/24 15:36, Simon Niederberger wrote:
> >> What if you create an empty jaxp.properties file and make it available to 
> >> the common ClassLoader (e.g. in lib/empty-jaxp.jar:/jaxp.properties) -- 
> >> does that prevent the problem?
>  >
> > I think that boils down to what I'm already doing with the system
> > property, getting to the FactoryFinder before it uses
> > findServiceProvider(). Haven't tried it, but I'm sure it would work.
>
> The difference is that Tomcat can ship it as a part of the distribution,
> while we shouldn't really be setting system properties like that. It
> might prevent, for example, applications using their own preferred
> implementation.
>
> > imho, Tomcat's EncodingDetector should init with either
> >
> > XML_INPUT_FACTORY = XMLInputFactory.newDefaultFactory();
> >
> > to just always use the JVM XMLInputFactory, or then
> >
> > XML_INPUT_FACTORY =
> > XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
> > EncodingDetector.class.getClassLoader());
> >
> > to honor it's own classloader (maybe
> > EncodingDetector.class.getClassLoader() is the wrong approach,
> > basically something getTomcatCommonClassloader())
>
> Yeah, that's probably better than an empty properties file wrapped in a
> JAR file wrapped in an enigma.
>
> We also have the option of using the JreMemoryLeakPreventionListener for
> this. There are already some XML-related protections in there, though
> this one is not specifically there.
>
> Do you have a preferred technique for fixing this? All of those
> suggestions seem equally reasonable. I think I have a slight preference
> for passing the JSP compiler's ClassLoader in to the factory method:
>
>XML_INPUT_FACTORY =
>  XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
>  EncodingDetector.class.getClassLoader());
>
> If you absolutey need to get your JSP compiler to have a custom
> XMLInputFactory, you can copy the JSP compiler into your application and
> it will use that ClassLoader instead.
>
> > What I just don't get is why there's so little online about others
> > havingEncodingDetector similar issues. Spring + libs like CXF or
> > jackson-dataformat-xml are common, both those libs have transitive
> > dependencies on woodstox-core. Throw in ehcache, also common, and your
> > webapp won't undeploy if it's the first webapp to load a JSP and thus
> > clinit EncodingDetector. Maybe the public has just given up on clean
> > undeploying.
>
> My guess is that most people don't really care too much about clean
> shutdowns of their applications. They either bring their applications up
> and down along with the whole JVM (either standalone or via Docker,
> etc.) or they use an embedded Tomcat where their own application hosts
> Tomcat and they start/stop that.
>
> Or they disable/ignore memory-leak detection log messages. *shrug*
>
> I wonder if there really are any use-cases for applications wanting
> Tomcat to specify the XMLInputFactory *to be used for JSP*. I suspect not.
>
> -chris
>
> > On Wed, Mar 20, 2024 at 7:01 PM Christopher Schultz
> >  wrote:
> >>
> >> Simon,
> >>
> >> On 3/20/24 09:59, Simon Niederberger wrote:
> >>> The whole thing is caused by Maven dependencies which pull in
> >>> com.fasterxml.woodstox:woodstox-core. The WstxInputFactory has a
> >>>
> >>> @ServiceProvider(XMLInputFactory.class)
> >>>
> >>> annotation, where ServiceProvider is
> >>> org.ehcache.spi.service.ServiceProvider. I didn't manage to trace the
> >>> key code section, but I do find that the
> >>> javax.xml.stream.FactoryFinder then ends up finding WstxInputFactory
> >>> as registered service provide

Re: Memory leak in EncodingDetector?

2024-03-21 Thread Christopher Schultz

Simon,

On 3/20/24 15:36, Simon Niederberger wrote:

What if you create an empty jaxp.properties file and make it available to the
common ClassLoader (e.g. in lib/empty-jaxp.jar:/jaxp.properties) -- does that
prevent the problem?

I think that boils down to what I'm already doing with the system
property, getting to the FactoryFinder before it uses
findServiceProvider(). Haven't tried it, but I'm sure it would work.

The difference is that Tomcat can ship it as a part of the distribution,
while we shouldn't really be setting system properties like that. It
might prevent, for example, applications using their own preferred
implementation.

imho, Tomcat's EncodingDetector should init with either

XML_INPUT_FACTORY = XMLInputFactory.newDefaultFactory();

to just always use the JVM XMLInputFactory, or then

XML_INPUT_FACTORY =
XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
EncodingDetector.class.getClassLoader());

to honor it's own classloader (maybe
EncodingDetector.class.getClassLoader() is the wrong approach,
basically something getTomcatCommonClassloader())

Yeah, that's probably better than an empty properties file wrapped in a
JAR file wrapped in an enigma.

We also have the option of using the JreMemoryLeakPreventionListener for
this. There are already some XML-related protections in there, though
this one is not specifically there.

Do you have a preferred technique for fixing this? All of those
suggestions seem equally reasonable. I think I have a slight preference
for passing the JSP compiler's ClassLoader in to the factory method:

XML_INPUT_FACTORY =
XMLInputFactory.newFactory(XMLInputFactory.class.getName(),
EncodingDetector.class.getClassLoader());

If you absolutey need to get your JSP compiler to have a custom
XMLInputFactory, you can copy the JSP compiler into your application and
it will use that ClassLoader instead.

What I just don't get is why there's so little online about others
havingEncodingDetector similar issues. Spring + libs like CXF or
jackson-dataformat-xml are common, both those libs have transitive
dependencies on woodstox-core. Throw in ehcache, also common, and your
webapp won't undeploy if it's the first webapp to load a JSP and thus
clinit EncodingDetector. Maybe the public has just given up on clean
undeploying.

My guess is that most people don't really care too much about clean
shutdowns of their applications. They either bring their applications up
and down along with the whole JVM (either standalone or via Docker,
etc.) or they use an embedded Tomcat where their own application hosts
Tomcat and they start/stop that.

Or they disable/ignore memory-leak detection log messages. *shrug*

I wonder if there really are any use-cases for applications wanting
Tomcat to specify the XMLInputFactory *to be used for JSP*. I suspect not.

-chris

On Wed, Mar 20, 2024 at 7:01 PM Christopher Schultz
wrote:

Simon,

On 3/20/24 09:59, Simon Niederberger wrote:

The whole thing is caused by Maven dependencies which pull in
com.fasterxml.woodstox:woodstox-core. The WstxInputFactory has a

@ServiceProvider(XMLInputFactory.class)

annotation, where ServiceProvider is
org.ehcache.spi.service.ServiceProvider. I didn't manage to trace the
key code section, but I do find that the
javax.xml.stream.FactoryFinder then ends up finding WstxInputFactory
as registered service provider. As WstxInputFactory is not on the
common classpath (it's in WEB-INF/lib), I assume it's the webapp
classloader which is used. Below is the stacktrace where
EncodingDetector clinit happens (I defined a
ch.want.funnel.FunnelApp$DelegatingXMLInputFactory to get
stacktraces):

Currently I'm setting

System.setProperty("javax.xml.stream.XMLInputFactory",
"com.sun.xml.internal.stream.XMLInputFactoryImpl");
to get a XMLInputFactory implementation which is on the common
loader's classpath, so the webapp can be undeployed cleanly.

So this works, right?

What if you create an empty jaxp.properties file and make it available
to the common ClassLoader (e.g. in lib/empty-jaxp.jar:/jaxp.properties)
-- does that prevent the problem?

I'm wondering if Tomcat should simply ship with an empty jaxp.properties
file to prevent this kind of thing from happening by default. If someone
wants to bundle an XMLInputFactory into Tomcat's lib/ directory and use
that, they could remove this file.

BTW that's an impressive stack trace. ;)

-chris

java.lang.RuntimeException: Stracktrace for tracking XMLInputFactory creation
at
ch.want.funnel.FunnelApp$DelegatingXMLInputFactory.(FunnelApp.java:107)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

Re: Memory leak in EncodingDetector?

2024-03-20 Thread Simon Niederberger

ore.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
> >  at 
> > org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109)
> >  at 
> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
> >  at 
> > org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:124)
> >  at 
> > org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:99)
> >  at 
> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> >  at 
> > org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
> >  at 
> > org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
> >  at 
> > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
> >  at 
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
> >  at 
> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
> >  at 
> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
> >  at 
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
> >  at 
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
> >  at 
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
> >  at 
> > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
> >  at 
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
> >  at 
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
> >  at 
> > org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:431)
> >  at 
> > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
> >  at 
> > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
> >  at 
> > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
> >  at 
> > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
> >  at 
> > org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> >  at 
> > org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> >  at 
> > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
> >  at java.base/java.lang.Thread.run(Thread.java:840)
> >
> >
> > Mühlegasse 18, 6340 Baar, Switzerland
> >
> > https://www.want.ch
> >
> > https://www.funnel.travel
> >
> >
> >
> >
> >
> > On Tue, Mar 19, 2024 at 12:31 PM Christopher Schultz
> >  wrote:
> >>
> >> Simon,
> >>
> >> On 3/18/24 15:17, Simon Niederberger wrote:
> >>> I'm analyzing a memory leak reported by Tomcat, and have narrowed it
> >>> down to org.apache.jasper.compiler.EncodingDetector:
> >>>
> >>> private static final XMLInputFactory XML_INPUT_FACTORY;
> >>> static {
> >>>   XML_INPUT_FACTORY = XMLInputFactory.newInstance();
> >>> }
> >>>
> >>> This class is called by webapp code on a GET request
> >>>
> >>>   at 
> >>> org.apache.jasper.compiler.EncodingDetector.(EncodingDetector.java:38)
> >>>   at 
> >

Re: Memory leak in EncodingDetector?

2024-03-20 Thread Christopher Schultz

rPageFilter.java:117)
 at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
 at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
 at 
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
 at 
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
 at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
 at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
 at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
 at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
 at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
 at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
 at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
 at 
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
 at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
 at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
 at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:431)
 at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
 at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
 at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
 at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
 at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
 at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
 at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
 at java.base/java.lang.Thread.run(Thread.java:840)


Mühlegasse 18, 6340 Baar, Switzerland

https://www.want.ch

https://www.funnel.travel





On Tue, Mar 19, 2024 at 12:31 PM Christopher Schultz
 wrote:


Simon,

On 3/18/24 15:17, Simon Niederberger wrote:

I'm analyzing a memory leak reported by Tomcat, and have narrowed it
down to org.apache.jasper.compiler.EncodingDetector:

private static final XMLInputFactory XML_INPUT_FACTORY;
static {
  XML_INPUT_FACTORY = XMLInputFactory.newInstance();
}

This class is called by webapp code on a GET request

  at 
org.apache.jasper.compiler.EncodingDetector.(EncodingDetector.java:38)
  at 
org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324)
  at 
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201)
  at 
org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128)
  at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:207)
  ...
  at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:396)
  at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)
  at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)
  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

The EncodingDetector class, if not yet loaded, will be loaded in the
common classloader, then continue by loading the XMLInputFactory using
the webapp context, and might end up with a XMLInputFactory
implementation from a webapp-provided JAR. If that happens, the webapp
can't undeploy. (In my case, woodstox WstxInputFactory registers
itself as ServiceProvider for XMLInputFactory)

For completeness: javax.xml.stream.FactoryFinder.findServiceProvider()
is called without classloader (cl = null), and has

if (cl == null) {
  //the current thread's context class loader
  serviceLoader = ServiceLoader.load(type);
} else {
  serviceLoader = ServiceLoader.load(type, cl);
}

I can't find anything online about memory leaks from webapp-provided
XMLInputFactory implementations, but this must be fairly common. Is my
understanding correct, or have I mis-configured something? (I'm mainly
wondering whether any XMLInputFactory-implementing JARs belong in
tomcat/lib, but again I'm not finding anything online confirming that)

Tomcat 10.1.19
JVM 17.0.10+7-Ubuntu-120.04.1
Ubuntu 20.04.6 LTS


I'm not sure how many web applications ship with an XMLInputSource, but
they definitely do exist. I'm fairly sure most applications won't set a
system property or ship with a stax.properties/jaxp.properties file to
override the default implementation, but of course if it's possible,
someone will eventually do it.

I'm curious: in your example, how are you declaring your

Re: Memory leak in EncodingDetector?

2024-03-20 Thread Simon Niederberger

dContextValve.invoke(StandardContextValve.java:90)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at 
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:431)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Thread.java:840)


Mühlegasse 18, 6340 Baar, Switzerland

https://www.want.ch

https://www.funnel.travel





On Tue, Mar 19, 2024 at 12:31 PM Christopher Schultz
 wrote:
>
> Simon,
>
> On 3/18/24 15:17, Simon Niederberger wrote:
> > I'm analyzing a memory leak reported by Tomcat, and have narrowed it
> > down to org.apache.jasper.compiler.EncodingDetector:
> >
> > private static final XMLInputFactory XML_INPUT_FACTORY;
> > static {
> >  XML_INPUT_FACTORY = XMLInputFactory.newInstance();
> > }
> >
> > This class is called by webapp code on a GET request
> >
> >  at 
> > org.apache.jasper.compiler.EncodingDetector.(EncodingDetector.java:38)
> >  at 
> > org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324)
> >  at 
> > org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201)
> >  at 
> > org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128)
> >  at 
> > org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:207)
> >  ...
> >  at 
> > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:396)
> >  at 
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)
> >  at 
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)
> >  at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
> >
> > The EncodingDetector class, if not yet loaded, will be loaded in the
> > common classloader, then continue by loading the XMLInputFactory using
> > the webapp context, and might end up with a XMLInputFactory
> > implementation from a webapp-provided JAR. If that happens, the webapp
> > can't undeploy. (In my case, woodstox WstxInputFactory registers
> > itself as ServiceProvider for XMLInputFactory)
> >
> > For completeness: javax.xml.stream.FactoryFinder.findServiceProvider()
> > is called without classloader (cl = null), and has
> >
> > if (cl == null) {
> >  //the current thread's context class loader
> >  serviceLoader = ServiceLoader.load(type);
> > } else {
> >  serviceLoader = ServiceLoader.load(type, cl);
> > }
> >
> > I can't find anything online about memory leaks from webapp-provided
> > XMLInputFactory implementations, but this must be fairly common. Is my
> > understanding correct, or have I mis-configured something? (I'm mainly
> > wondering whether any XMLInputFactory-implementing JARs belong in
> > tomcat/lib, but again I'm not finding anything online confirming that)
> >
> > Tomcat 10.1.19
> > JVM 17.0.10+7-Ubuntu-120.04.1
> > Ubuntu 20.04.6 LTS
>
> I'm not sure how many web applications ship with an XMLInputSource, but
> they definitely do exist. I'm fairly sure most applications won't set a
> system property or ship with a stax.properties/jaxp.properties file to
> override the default implementation, but of course if it's possible,
> someone will eventually do it.
>
> I'm curious: in your example, how are you declaring your implementation
> class, and which implementation are you using?
>
> Are you able to log in EncodingDetector. what the value of the
>

Re: Memory leak in EncodingDetector?

2024-03-19 Thread Christopher Schultz


Simon,

On 3/18/24 15:17, Simon Niederberger wrote:

I'm analyzing a memory leak reported by Tomcat, and have narrowed it
down to org.apache.jasper.compiler.EncodingDetector:

private static final XMLInputFactory XML_INPUT_FACTORY;
static {
 XML_INPUT_FACTORY = XMLInputFactory.newInstance();
}

This class is called by webapp code on a GET request

 at 
org.apache.jasper.compiler.EncodingDetector.(EncodingDetector.java:38)
 at 
org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324)
 at 
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201)
 at 
org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128)
 at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:207)
 ...
 at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:396)
 at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)
 at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

The EncodingDetector class, if not yet loaded, will be loaded in the
common classloader, then continue by loading the XMLInputFactory using
the webapp context, and might end up with a XMLInputFactory
implementation from a webapp-provided JAR. If that happens, the webapp
can't undeploy. (In my case, woodstox WstxInputFactory registers
itself as ServiceProvider for XMLInputFactory)

For completeness: javax.xml.stream.FactoryFinder.findServiceProvider()
is called without classloader (cl = null), and has

if (cl == null) {
 //the current thread's context class loader
 serviceLoader = ServiceLoader.load(type);
} else {
 serviceLoader = ServiceLoader.load(type, cl);
}

I can't find anything online about memory leaks from webapp-provided
XMLInputFactory implementations, but this must be fairly common. Is my
understanding correct, or have I mis-configured something? (I'm mainly
wondering whether any XMLInputFactory-implementing JARs belong in
tomcat/lib, but again I'm not finding anything online confirming that)

Tomcat 10.1.19
JVM 17.0.10+7-Ubuntu-120.04.1
Ubuntu 20.04.6 LTS


I'm not sure how many web applications ship with an XMLInputSource, but 
they definitely do exist. I'm fairly sure most applications won't set a 
system property or ship with a stax.properties/jaxp.properties file to 
override the default implementation, but of course if it's possible, 
someone will eventually do it.


I'm curious: in your example, how are you declaring your implementation 
class, and which implementation are you using?


Are you able to log in EncodingDetector. what the value of the 
thread's context classloader is? I would expect that it's using the 
common classloader, as you say, and that the implementation class would 
also be loaded using that same classloader.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Memory leak in EncodingDetector?

2024-03-18 Thread Simon Niederberger

Hi

I'm analyzing a memory leak reported by Tomcat, and have narrowed it
down to org.apache.jasper.compiler.EncodingDetector:

private static final XMLInputFactory XML_INPUT_FACTORY;
static {
XML_INPUT_FACTORY = XMLInputFactory.newInstance();
}

This class is called by webapp code on a GET request

at 
org.apache.jasper.compiler.EncodingDetector.(EncodingDetector.java:38)
at 
org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324)
at 
org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201)
at 
org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128)
at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:207)
...
at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:396)
at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

The EncodingDetector class, if not yet loaded, will be loaded in the
common classloader, then continue by loading the XMLInputFactory using
the webapp context, and might end up with a XMLInputFactory
implementation from a webapp-provided JAR. If that happens, the webapp
can't undeploy. (In my case, woodstox WstxInputFactory registers
itself as ServiceProvider for XMLInputFactory)

For completeness: javax.xml.stream.FactoryFinder.findServiceProvider()
is called without classloader (cl = null), and has

if (cl == null) {
//the current thread's context class loader
serviceLoader = ServiceLoader.load(type);
} else {
serviceLoader = ServiceLoader.load(type, cl);
}

I can't find anything online about memory leaks from webapp-provided
XMLInputFactory implementations, but this must be fairly common. Is my
understanding correct, or have I mis-configured something? (I'm mainly
wondering whether any XMLInputFactory-implementing JARs belong in
tomcat/lib, but again I'm not finding anything online confirming that)

Tomcat 10.1.19
JVM 17.0.10+7-Ubuntu-120.04.1
Ubuntu 20.04.6 LTS



Simon

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: Tomcat 9.0.76 Memory leak with Java 17

2023-07-13 Thread Christopher Schultz


Thomas,

On 7/13/23 02:19, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello,


-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Mittwoch, 12. Juli 2023 21:34
An: users@tomcat.apache.org
Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17

Michael,

On 7/12/23 07:33, Michael Osipov wrote:

On 2023/07/11 18:16:24 Christopher Schultz wrote:

You should report all of the previous issues to Oracle against their
ORDS version 22.1 and ask them to fix them. It's why you write those
big, fat checks in the first place ;)


This doesn't really matter. I have reported a memory leak in OJDBC
many years ago where a background thread pins the WebappClassLoader.
Answer from Oracle: This happens only once, does not repeat. We won't
address that...

Fair enough, but that doesn't mean it's not worth reporting.

BTW the fine folks at Sun/Oracle and Connector/J are similarly confused
when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had a
few different patches that all waved their hands _still_ failing to actually
understand and fix the problem.

-chris



It's not just Oracle JDBC driver.
Same can happen with SQL-Server JDBC driver when you use Kerberos Auth.
Ticket was also closed without a fix.


Yeah, and I think it just comes down to failure to understand the problem.

The problem with Connector/J has always been that they launch a 
background thread to clean things up. All they have to do is provide a 
method to "shut down background threads".


I kept getting pushback like "they are daemon threads, they won't keep 
your JVM running when your application stops" and I was like "hey 
sometimes the application stops and restarts without shutting down the 
JVM #webapps" and they were like "what does that even mean CLOSED WONTFIX".


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AW: Tomcat 9.0.76 Memory leak with Java 17

2023-07-13 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello,

> -Ursprüngliche Nachricht-
> Von: Christopher Schultz 
> Gesendet: Mittwoch, 12. Juli 2023 21:34
> An: users@tomcat.apache.org
> Betreff: Re: Tomcat 9.0.76 Memory leak with Java 17
> 
> Michael,
> 
> On 7/12/23 07:33, Michael Osipov wrote:
> > On 2023/07/11 18:16:24 Christopher Schultz wrote:
> >> You should report all of the previous issues to Oracle against their
> >> ORDS version 22.1 and ask them to fix them. It's why you write those
> >> big, fat checks in the first place ;)
> >
> > This doesn't really matter. I have reported a memory leak in OJDBC
> > many years ago where a background thread pins the WebappClassLoader.
> > Answer from Oracle: This happens only once, does not repeat. We won't
> > address that...
> Fair enough, but that doesn't mean it's not worth reporting.
> 
> BTW the fine folks at Sun/Oracle and Connector/J are similarly confused
> when it comes to ClassLoader pinning in /their/ JDBC driver. I think they had 
> a
> few different patches that all waved their hands _still_ failing to actually
> understand and fix the problem.
> 
> -chris
> 

It's not just Oracle JDBC driver.
Same can happen with SQL-Server JDBC driver when you use Kerberos Auth. 
Ticket was also closed without a fix.

RE: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread James Boggs

Chris,

Yes it is unintentional. Actually once we start it with the Windows service, 
and run through a few reports on the website, it stops in just ba few minutes.
We will look at the java heap size settings.

Regards,

James Boggs


-Original Message-
From: Christopher Schultz  
Sent: Wednesday, July 12, 2023 3:55 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17

James,

On 7/12/23 15:41, James Boggs wrote:
> Thanks for the input. I will forward the email to our developers to 
> look at the heap size settings being different.
> 
> We have a Windows service that is used to start/stop Tomcat. When this 
> happens we find that the Windows service is no longer running.
Depending upon your settings, you may have more than one log file to look at. 
Look at everything in CATALINA_BASE/logs the next time the service goes down to 
see if there is a clue in any of them.

You could also look for hs_pid[process-id] files (or the Windows equivalent, if 
it's different) that are usually dumped when a JVM crash occurs.

Each of your logs seems to show that Tomcat is being shut-down in an orderly 
way, though. Are you sure it's crashing and not being intentionally stopped?

-chris

> -Original Message-
> From: Christopher Schultz 
> Sent: Wednesday, July 12, 2023 3:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.76 Memory leak with Java 17
> 
> Suvendu,
> 
> On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
>> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz 
>>  wrote:
>>>
>>> James,
>>>
>>> On 7/11/23 10:21, James Boggs wrote:
>>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on 
>>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>>
>>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 
>>>> and to ORDS 22.1, then used Java 17 to create a new Java Keystore 
>>>> and a new SSL csr file, and imported a new SSL certificate from the 
>>>> CA into the new keystore.
>>>>
>>>> The website works but after logging in there are memory leak 
>>>> warnings and the Tomcat service crashes within just a couple of minutes.
>>>>
>>>> We even upgraded to 9.0.76 and the issue persists.
>>>>
>>>> Below is an excerpt from the stderr log file.
>>>>
>>>> I have been unable to find any recent threads on this, any help is 
>>>> appreciated. Is any other information needed?
>>>
>>> I think you have included all necessary information. I'm 
>>> chopping-out the irrrelevant bits:
>>>
>>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to 
>>>> unregister it when the web application was stopped. To prevent a 
>>>> memory leak, the JDBC Driver has been forcibly unregistered.
>>>
>>> This is actually "okay" in that Tomcat has detected a global JDBC 
>>> driver registration performed by the application, and is fixing the 
>>> problem for you. The application, however, is making a mistake by 
>>> not de-registering that JDBC driver. Your options are to move the 
>>> driver library from your application into Tomcat's lib/ directory, 
>>> fix the application to de-register the driver when it shuts down, or just 
>>> ignore the warning.
>>> But you should fix the application.
>>>
>>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>>> appears to have started a thread named 
>>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] 
>>>> but has failed to stop it. This is very likely to create a memory leak. 
>>>> Stack trace of thread:
>>>
>>> There are multiple instances of this same message and THIS is your 
>>> problem. The problem is what the error message says: your 
>>> application has started a Thread and never stopped it. The "memory 
>>> leak" comes from the fact that the Thread has inherited the web 
>>> application's ClassLoader and the web application is being 
>>> re-loaded. When that happens, Tomcat discards the ClassLoader which 
>>> usually means the GC will clean up after it at some point in the 
>>> future. But that Thread is still running and will keep the ClassLoader in 
>>> memory, likely forever.
>>>
>>> You have a few options:
>>>
>>> 1. Fix the application. The application needs to shut-down any 
>>> threads is

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread Christopher Schultz


James,

On 7/12/23 15:41, James Boggs wrote:

Thanks for the input. I will forward the email to our developers to
look at the heap size settings being different.

We have a Windows service that is used to start/stop Tomcat. When
this happens we find that the Windows service is no longer running.
Depending upon your settings, you may have more than one log file to 
look at. Look at everything in CATALINA_BASE/logs the next time the 
service goes down to see if there is a clue in any of them.


You could also look for hs_pid[process-id] files (or the Windows 
equivalent, if it's different) that are usually dumped when a JVM crash 
occurs.


Each of your logs seems to show that Tomcat is being shut-down in an 
orderly way, though. Are you sure it's crashing and not being 
intentionally stopped?


-chris


-Original Message-
From: Christopher Schultz 
Sent: Wednesday, July 12, 2023 3:32 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17

Suvendu,

On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:

On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
 wrote:


James,

On 7/11/23 10:21, James Boggs wrote:

We had a stable SSL enabled website with Apache Tomcat 9.0.73 on
Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.

We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and
to ORDS 22.1, then used Java 17 to create a new Java Keystore and a
new SSL csr file, and imported a new SSL certificate from the CA
into the new keystore.

The website works but after logging in there are memory leak
warnings and the Tomcat service crashes within just a couple of minutes.

We even upgraded to 9.0.76 and the issue persists.

Below is an excerpt from the stderr log file.

I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?


I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:


2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a
memory leak, the JDBC Driver has been forcibly unregistered.


This is actually "okay" in that Tomcat has detected a global JDBC
driver registration performed by the application, and is fixing the
problem for you. The application, however, is making a mistake by not
de-registering that JDBC driver. Your options are to move the driver
library from your application into Tomcat's lib/ directory, fix the
application to de-register the driver when it shuts down, or just ignore the 
warning.
But you should fix the application.


2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but 
has failed to stop it. This is very likely to create a memory leak. Stack trace 
of thread:


There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes
from the fact that the Thread has inherited the web application's
ClassLoader and the web application is being re-loaded. When that
happens, Tomcat discards the ClassLoader which usually means the GC
will clean up after it at some point in the future. But that Thread
is still running and will keep the ClassLoader in memory, likely forever.

You have a few options:

1. Fix the application. The application needs to shut-down any
threads is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.

2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the
JVM, anyway.



I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.


2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler 
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds
2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler 
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:40.495Z INFOStopping service [Catalina]
2023-07-10T21:35:40.910Z INFODestroyed pool : 
|default|lo|-2023-07-10T20-51-53.099285500Z


Good point: Tomcat will complain about this stuff on shutdown even if the 
warnings are irrelevant because the application-stop leak-checks are run 
whether Tomcat is going to stop or not.

So operationally, these warnings may be irrelevant.

(IMHO you should still fix the application!)


Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting

RE: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread James Boggs

Thanks for the input. I will forward the email to our developers to look at the 
heap size settings being different.
We have a Windows service that is used to start/stop Tomcat. When this happens 
we find that the Windows service is no longer running.

Thanks,

James Boggs


-Original Message-
From: Christopher Schultz  
Sent: Wednesday, July 12, 2023 3:32 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.76 Memory leak with Java 17

Suvendu,

On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:
> On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz 
>  wrote:
>>
>> James,
>>
>> On 7/11/23 10:21, James Boggs wrote:
>>> We had a stable SSL enabled website with Apache Tomcat 9.0.73 on 
>>> Windows Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
>>>
>>> We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and 
>>> to ORDS 22.1, then used Java 17 to create a new Java Keystore and a 
>>> new SSL csr file, and imported a new SSL certificate from the CA 
>>> into the new keystore.
>>>
>>> The website works but after logging in there are memory leak 
>>> warnings and the Tomcat service crashes within just a couple of minutes.
>>>
>>> We even upgraded to 9.0.76 and the issue persists.
>>>
>>> Below is an excerpt from the stderr log file.
>>>
>>> I have been unable to find any recent threads on this, any help is 
>>> appreciated. Is any other information needed?
>>
>> I think you have included all necessary information. I'm chopping-out 
>> the irrrelevant bits:
>>
>>> 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
>>> registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to 
>>> unregister it when the web application was stopped. To prevent a 
>>> memory leak, the JDBC Driver has been forcibly unregistered.
>>
>> This is actually "okay" in that Tomcat has detected a global JDBC 
>> driver registration performed by the application, and is fixing the 
>> problem for you. The application, however, is making a mistake by not 
>> de-registering that JDBC driver. Your options are to move the driver 
>> library from your application into Tomcat's lib/ directory, fix the 
>> application to de-register the driver when it shuts down, or just ignore the 
>> warning.
>> But you should fix the application.
>>
>>> 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
>>> appears to have started a thread named 
>>> [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] 
>>> but has failed to stop it. This is very likely to create a memory leak. 
>>> Stack trace of thread:
>>
>> There are multiple instances of this same message and THIS is your 
>> problem. The problem is what the error message says: your application 
>> has started a Thread and never stopped it. The "memory leak" comes 
>> from the fact that the Thread has inherited the web application's 
>> ClassLoader and the web application is being re-loaded. When that 
>> happens, Tomcat discards the ClassLoader which usually means the GC 
>> will clean up after it at some point in the future. But that Thread 
>> is still running and will keep the ClassLoader in memory, likely forever.
>>
>> You have a few options:
>>
>> 1. Fix the application. The application needs to shut-down any 
>> threads is starts during its operation, preferrably in a 
>> ServletContextListener's destroy method or similar.
>>
>> 2. Don't hot-reload your application. Instead, shut-down the JVM and 
>> re-start it. Ovbviously, this may have availability implications, but 
>> then again so does running out of memory and having to bounce the 
>> JVM, anyway.
>>
> 
> I was checking the stacks and saw this: They might not be doing 
> "hot-deploy" of the app. AFAIK those messages come once someone stops 
> Tomcat.
> 
>> 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler 
>> ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds
>> 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler 
>> ["https-openssl-nio-10.2.251.132-443"]
>> 2023-07-10T21:35:40.495Z INFOStopping service [Catalina]
>> 2023-07-10T21:35:40.910Z INFODestroyed pool : 
>> |default|lo|-2023-07-10T20-51-53.099285500Z

Good point: Tomcat will complain about this stuff on shutdown even if the 
warnings are irrelevant because the application-stop leak-c

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread Christopher Schultz


Michael,

On 7/12/23 07:33, Michael Osipov wrote:

On 2023/07/11 18:16:24 Christopher Schultz wrote:

You should report all of the previous issues to Oracle against their
ORDS version 22.1 and ask them to fix them. It's why you write those
big, fat checks in the first place ;)


This doesn't really matter. I have reported a memory leak in OJDBC
many years ago where a background thread pins the WebappClassLoader.
Answer from Oracle: This happens only once, does not repeat. We won't
address that...

Fair enough, but that doesn't mean it's not worth reporting.

BTW the fine folks at Sun/Oracle and Connector/J are similarly confused 
when it comes to ClassLoader pinning in /their/ JDBC driver. I think 
they had a few different patches that all waved their hands _still_ 
failing to actually understand and fix the problem.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread Christopher Schultz


Suvendu,

On 7/12/23 07:11, Suvendu Sekhar Mondal wrote:

On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
 wrote:


James,

On 7/11/23 10:21, James Boggs wrote:

We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.

We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
csr file, and imported a new SSL certificate from the CA into the new
keystore.

The website works but after logging in there are memory leak warnings
and the Tomcat service crashes within just a couple of minutes.

We even upgraded to 9.0.76 and the issue persists.

Below is an excerpt from the stderr log file.

I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?


I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:


2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a memory
leak, the JDBC Driver has been forcibly unregistered.


This is actually "okay" in that Tomcat has detected a global JDBC driver
registration performed by the application, and is fixing the problem for
you. The application, however, is making a mistake by not de-registering
that JDBC driver. Your options are to move the driver library from your
application into Tomcat's lib/ directory, fix the application to
de-register the driver when it shuts down, or just ignore the warning.
But you should fix the application.


2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but 
has failed to stop it. This is very likely to create a memory leak. Stack trace 
of thread:


There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes from
the fact that the Thread has inherited the web application's ClassLoader
and the web application is being re-loaded. When that happens, Tomcat
discards the ClassLoader which usually means the GC will clean up after
it at some point in the future. But that Thread is still running and
will keep the ClassLoader in memory, likely forever.

You have a few options:

1. Fix the application. The application needs to shut-down any threads
is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.

2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the JVM,
anyway.



I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.


2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler 
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds
2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler 
["https-openssl-nio-10.2.251.132-443"]
2023-07-10T21:35:40.495Z INFOStopping service [Catalina]
2023-07-10T21:35:40.910Z INFODestroyed pool : 
|default|lo|-2023-07-10T20-51-53.099285500Z


Good point: Tomcat will complain about this stuff on shutdown even if 
the warnings are irrelevant because the application-stop leak-checks are 
run whether Tomcat is going to stop or not.


So operationally, these warnings may be irrelevant.

(IMHO you should still fix the application!)


Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.


10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dconfig.url=D:\ords222
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xms1024M
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xmx1024M



10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
exit
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
abort
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xms128m
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xmx256m


Oh,

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread Michael Osipov

On 2023/07/11 18:16:24 Christopher Schultz wrote:
> You should report all of the previous issues to Oracle against their 
> ORDS version 22.1 and ask them to fix them. It's why you write those 
> big, fat checks in the first place ;)

This doesn't really matter. I have reported a memory leak in OJDBC many years 
ago where a background thread pins the WebappClassLoader. Answer from Oracle: 
This happens only once, does not repeat. We won't address that...

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-12 Thread Suvendu Sekhar Mondal

Hi Chris,

On Tue, Jul 11, 2023 at 11:48 PM Christopher Schultz
 wrote:
>
> James,
>
> On 7/11/23 10:21, James Boggs wrote:
> > We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
> > Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
> >
> > We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
> > ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
> > csr file, and imported a new SSL certificate from the CA into the new
> > keystore.
> >
> > The website works but after logging in there are memory leak warnings
> > and the Tomcat service crashes within just a couple of minutes.
> >
> > We even upgraded to 9.0.76 and the issue persists.
> >
> > Below is an excerpt from the stderr log file.
> >
> > I have been unable to find any recent threads on this, any help is
> > appreciated. Is any other information needed?
>
> I think you have included all necessary information. I'm chopping-out
> the irrrelevant bits:
>
> > 2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
> > registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
> > unregister it when the web application was stopped. To prevent a memory
> > leak, the JDBC Driver has been forcibly unregistered.
>
> This is actually "okay" in that Tomcat has detected a global JDBC driver
> registration performed by the application, and is fixing the problem for
> you. The application, however, is making a mistake by not de-registering
> that JDBC driver. Your options are to move the driver library from your
> application into Tomcat's lib/ directory, fix the application to
> de-register the driver when it shuts down, or just ignore the warning.
> But you should fix the application.
>
> > 2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
> > appears to have started a thread named
> > [oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] 
> > but has failed to stop it. This is very likely to create a memory leak. 
> > Stack trace of thread:
>
> There are multiple instances of this same message and THIS is your
> problem. The problem is what the error message says: your application
> has started a Thread and never stopped it. The "memory leak" comes from
> the fact that the Thread has inherited the web application's ClassLoader
> and the web application is being re-loaded. When that happens, Tomcat
> discards the ClassLoader which usually means the GC will clean up after
> it at some point in the future. But that Thread is still running and
> will keep the ClassLoader in memory, likely forever.
>
> You have a few options:
>
> 1. Fix the application. The application needs to shut-down any threads
> is starts during its operation, preferrably in a
> ServletContextListener's destroy method or similar.
>
> 2. Don't hot-reload your application. Instead, shut-down the JVM and
> re-start it. Ovbviously, this may have availability implications, but
> then again so does running out of memory and having to bounce the JVM,
> anyway.
>

I was checking the stacks and saw this: They might not be doing
"hot-deploy" of the app. AFAIK those messages come once someone stops
Tomcat.

> 2023-07-10T20:52:06.292Z INFOStarting ProtocolHandler 
> ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T20:52:06.346Z INFOServer startup in [28980] milliseconds
> 2023-07-10T21:35:40.487Z INFOPausing ProtocolHandler 
> ["https-openssl-nio-10.2.251.132-443"]
> 2023-07-10T21:35:40.495Z INFOStopping service [Catalina]
> 2023-07-10T21:35:40.910Z INFODestroyed pool : 
> |default|lo|-2023-07-10T20-51-53.099285500Z

Another thing I noticed was the difference in allocated Java heap. It
appears that first it sets min=max=1GB and after that it's setting max
heap to 256MB. That could be the problem. JVM might be crashing
because of heap shortage.

> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Dconfig.url=D:\ords222
> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xms1024M
> 10-Jul-2023 16:51:35.481 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> -Xmx1024M

> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> exit
> 10-Jul-2023 16:51:35.486 INFO [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
> abort
> 10-Jul-2023 16:51:35.486 INFO [main] 
>

Re: Tomcat 9.0.76 Memory leak with Java 17

2023-07-11 Thread Christopher Schultz

James,

On 7/11/23 10:21, James Boggs wrote:
We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows
Server 2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.

We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to
ORDS 22.1, then used Java 17 to create a new Java Keystore and a new SSL
csr file, and imported a new SSL certificate from the CA into the new
keystore.

The website works but after logging in there are memory leak warnings
and the Tomcat service crashes within just a couple of minutes.

We even upgraded to 9.0.76 and the issue persists.

Below is an excerpt from the stderr log file.

I have been unable to find any recent threads on this, any help is
appreciated. Is any other information needed?

I think you have included all necessary information. I'm chopping-out
the irrrelevant bits:

2023-07-10T21:35:40.939Z WARNING The web application [rplans-vpd]
registered the JDBC driver [oracle.jdbc.OracleDriver] but failed to
unregister it when the web application was stopped. To prevent a memory
leak, the JDBC Driver has been forcibly unregistered.

This is actually "okay" in that Tomcat has detected a global JDBC driver
registration performed by the application, and is fixing the problem for
you. The application, however, is making a mistake by not de-registering
that JDBC driver. Your options are to move the driver library from your
application into Tomcat's lib/ directory, fix the application to
de-register the driver when it shuts down, or just ignore the warning.
But you should fix the application.

2023-07-10T21:35:40.944Z WARNING The web application [rplans-vpd]
appears to have started a thread named
[oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

There are multiple instances of this same message and THIS is your
problem. The problem is what the error message says: your application
has started a Thread and never stopped it. The "memory leak" comes from
the fact that the Thread has inherited the web application's ClassLoader
and the web application is being re-loaded. When that happens, Tomcat
discards the ClassLoader which usually means the GC will clean up after
it at some point in the future. But that Thread is still running and
will keep the ClassLoader in memory, likely forever.

You have a few options:

1. Fix the application. The application needs to shut-down any threads
is starts during its operation, preferrably in a
ServletContextListener's destroy method or similar.

2. Don't hot-reload your application. Instead, shut-down the JVM and
re-start it. Ovbviously, this may have availability implications, but
then again so does running out of memory and having to bounce the JVM,
anyway.

2023-07-10T21:35:40.944Z SEVERE The web application [rplans-vpd]
created a ThreadLocal with key of type
[java.lang.ThreadLocal.SuppliedThreadLocal] (value
[java.lang.ThreadLocal$SuppliedThreadLocal@427241b4]) and a value of
type [oracle.ucp.common.Core.Match] (value [UNKNOWN]) but failed to
remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.

This is actually "okay" in that Tomcat has detected your application's
ThreadLocal variables (objects bound to one or more Threads which are
owned by the application) which are not being cleaned-up by the
application, and it's fixing the problem for you by, over time, killing
each of those Threads and replacing them in the Thread pool for you.
Your options are to fix the application or to ignore the warning. But
you should fix the application.

It appears that your upgrade of ORDS has introduced a lot of stuff that
doesn't play well with hot-reloading of the application. I'm assuming
that you aren't responsible for maintaining ORDS... Oracle is.

You should report all of the previous issues to Oracle against their
ORDS version 22.1 and ask them to fix them. It's why you write those
big, fat checks in the first place ;)

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 9.0.76 Memory leak with Java 17

2023-07-11 Thread James Boggs

Hi,

We had a stable SSL enabled website with Apache Tomcat 9.0.73 on Windows Server 
2012 o/s, Java 8, Oracle ORDS 21.4 and SSL.
We simultaneously upgraded to Tomcat 9.0.75, upgraded to Java 17 and to ORDS 
22.1, then used Java 17 to create a new Java Keystore and a new SSL csr file, 
and imported a new SSL certificate from the CA into the new keystore.
The website works but after logging in there are memory leak warnings and the 
Tomcat service crashes within just a couple of minutes.
We even upgraded to 9.0.76 and the issue persists.
Below is an excerpt from the stderr log file.
I have been unable to find any recent threads on this, any help is appreciated. 
Is any other information needed?

-- start of logfile

2023-07-10 16:51:33 Apache Commons Daemon procrun stderr initialized.
10-Jul-2023 16:51:35.434 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version name:   
Apache Tomcat/9.0.76
10-Jul-2023 16:51:35.449 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server built:  
Jun 5 2023 07:17:04 UTC
10-Jul-2023 16:51:35.451 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version number: 
9.0.76.0
10-Jul-2023 16:51:35.451 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Name:   
Windows Server 2012 R2
10-Jul-2023 16:51:35.451 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Version:6.3
10-Jul-2023 16:51:35.451 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Architecture:  
amd64
10-Jul-2023 16:51:35.452 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Java Home: 
D:\JAVA\JDK
10-Jul-2023 16:51:35.452 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:   
17.0.7+8-LTS-224
10-Jul-2023 16:51:35.452 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
Oracle Corporation
10-Jul-2023 16:51:35.452 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: 
D:\tomcat9
10-Jul-2023 16:51:35.454 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: 
D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dconfig.url=D:\ords222
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xms1024M
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xmx1024M
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dcatalina.home=D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dcatalina.base=D:\tomcat9
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dignore.endorsed.dirs=D:\tomcat9\endorsed
10-Jul-2023 16:51:35.481 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.io.tmpdir=D:\tomcat9\temp
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.config.file=D:\tomcat9\conf\logging.properties
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.lang=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.io=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.util=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
exit
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
abort
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xms128m
10-Jul-2023 16:51:35.486 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xmx256m
10-Jul-2023 16:51:35.496 INFO [main

Re: Tomcat / Oracle JDBC OracleTimeoutPollingThread memory leak?

2023-02-15 Thread Mark Thomas


On 13/02/2023 19:12, Ragosta, Vincent wrote:




Is the above GC root, an example of the above condition?


Yes.


If so, can this be mitigated using the JreMemoryLeakPreventionListener, as 
illustrated in the Tomcat documentation, here -- 
https://tomcat.apache.org/tomcat-9.0-doc/config/listeners.html#JreMemoryLeakPreventionListener_Examples?


Yes.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat / Oracle JDBC OracleTimeoutPollingThread memory leak?

2023-02-13 Thread Ragosta, Vincent

Our Tomcat instance is configured to create a heap dump on out of memory 
conditions, and we often see a large amount of memory allocated to 
OracleTimeoutPollingThread. For example, here is once such instance:

465,922Kb (57.4%) Object tree for GC root(s) Java Static 
oracle.jdbc.driver.OracleTimeoutThreadPerVM.watchdog Leak candidate(s) found
Root object 
oracle.jdbc.driver.OracleTimeoutPollingThread@c492d2c0oracle.jdbc.driver.OracleTimeoutPollingThread(name
 : "OracleTimeoutPollingThread", priority : 10, threadQ : null, eetop : 
392972288, single_step : false, daemon : true, stillborn : false, target : 
null, group : j.l.ThreadGroup@c02fa828, 
contextClassLoader : 
org.apache.catalina.loader.ParallelWebappClassLoader@c02fb4c8,
 inheritedAccessControlContext : 
java.security.AccessControlContext@c492d448,
 threadLocals : null, inheritableThreadLocals : 
j.l.ThreadLocal$ThreadLocalMap(size: 2), stackSize : 0, nativeParkEventPointer 
: 0, tid : 119, threadStatus : 225, parkBlocker : null, blocker : null, 
blockerLock : Object@c492d660, uncaughtExceptionHandler : null, 
threadLocalRandomSeed : 0, threadLocalRandomProbe : 0, 
threadLocalRandomSecondarySeed : 0, knownTimeouts : 
oracle.jdbc.driver.OracleTimeoutThreadPerVM[](size: 32), count : 20, 
sleepMillis : 1000)

  1.  oracle.jdbc.driver.OracleTimeoutPollingThread ↘465,922Kb (57.4%), self 
136b (< 0.1%), 1 object(s)

I came across the following note:

Oracle JDBC driver, at least in some versions (such as ojdbc6), have a timeout 
detection thread, _oracle.jdbc.driver.OracleTimeoutPollingThread_, that 
according to reports has as its context classloader the classloader of the web 
application from which the first JDBC connection is requested, even if the 
driver itself resides on the server level. It seem this can be prevented by 
loading the class _oracle.jdbc.driver.OracleTimeoutThreadPerVM_ using the 
system classloader before any JDBC connections are opened, which can be 
achieved in _contextInitialized()_ of our 
_ServletContextListener_.
 (Thanks to Hal Deadman for the report!)

Ref: https://java.jiderhamn.se/category/classloader-leaks/

Is the above GC root, an example of the above condition?

We are using the following:

Oracle JDBC driver version 18.3.0.0.0

Tomcat 9.0.60.



If so, can this be mitigated using the JreMemoryLeakPreventionListener, as 
illustrated in the Tomcat documentation, here -- 
https://tomcat.apache.org/tomcat-9.0-doc/config/listeners.html#JreMemoryLeakPreventionListener_Examples?



Thank you,

Vincent

RE: Tomcat 9.0.65 suspected memory leak

2023-02-10 Thread Chen Levy

Thanks Mark, workaround seem to be working
Chen

> -Original Message-
> From: Mark Thomas 
> Sent: Thursday, February 9, 2023 12:41
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.65 suspected memory leak
> 
> On 09/02/2023 13:25, Mark Thomas wrote:
> > On 09/02/2023 13:04, Mark Thomas wrote:
> >> On 04/02/2023 22:06, Chen Levy wrote:
> >>
> >>> Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate
> >>> open connections:
> >>> I took a fresh Tomcat, unzipped and modified server.xml with only
> >>> the
> >>> following:
> >>> 1. Changed port 8080 to port 80
> >>> 2. Changed port 8443 to port 443
> >>> 3. Uncommented the nio connector and added the snippet
> >>>      >>> className="org.apache.coyote.http2.Http2Protocol" />
> >>>  
> >>>   >>> certificateKeystoreFile="conf/tomcat_noroot.p12"
> >>>   certificateKeyAlias="..."
> >>>   certificateKeystorePassword="..."
> >>>   certificateKeystoreType="PKCS12"/>
> >>>  
> >>>
> >>> I used Chrome to call the default index.html with Wireshark in the
> >>> middle:
> >>> With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY
> >>> from the server.
> >>> With 9.0.65 - No GOAWAY was sent, and the server and client kept
> >>> ACKing each other.
> >>>
> >>> Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.
> >>>
> >>> Test was conducted with:
> >>> Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23) Chrome Version
> >>> 109.0.5414.120 JDK 17.0.6+10 Windows 11
> >>
> >> Thanks for the reproduction details. I'll take a look now.
> >
> > A quick workaround is to configure useAsyncIO="false" on the Connector.
> 
> Fixed for the next round of releases.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.65 suspected memory leak

2023-02-09 Thread Mark Thomas


On 09/02/2023 13:25, Mark Thomas wrote:

On 09/02/2023 13:04, Mark Thomas wrote:

On 04/02/2023 22:06, Chen Levy wrote:

Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate 
open connections:
I took a fresh Tomcat, unzipped and modified server.xml with only the 
following:

1. Changed port 8080 to port 80
2. Changed port 8443 to port 443
3. Uncommented the nio connector and added the snippet
    className="org.apache.coyote.http2.Http2Protocol" />

 
 certificateKeystoreFile="conf/tomcat_noroot.p12"

  certificateKeyAlias="..."
  certificateKeystorePassword="..."
  certificateKeystoreType="PKCS12"/>
 

I used Chrome to call the default index.html with Wireshark in the 
middle:
With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY 
from the server.
With 9.0.65 - No GOAWAY was sent, and the server and client kept 
ACKing each other.


Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.

Test was conducted with:
Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23)
Chrome Version 109.0.5414.120
JDK 17.0.6+10
Windows 11


Thanks for the reproduction details. I'll take a look now.


A quick workaround is to configure useAsyncIO="false" on the Connector.


Fixed for the next round of releases.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.65 suspected memory leak

2023-02-09 Thread Mark Thomas


On 09/02/2023 13:04, Mark Thomas wrote:

On 04/02/2023 22:06, Chen Levy wrote:

Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate open 
connections:
I took a fresh Tomcat, unzipped and modified server.xml with only the 
following:

1. Changed port 8080 to port 80
2. Changed port 8443 to port 443
3. Uncommented the nio connector and added the snippet
    className="org.apache.coyote.http2.Http2Protocol" />

 
 certificateKeystoreFile="conf/tomcat_noroot.p12"

  certificateKeyAlias="..."
  certificateKeystorePassword="..."
  certificateKeystoreType="PKCS12"/>
 

I used Chrome to call the default index.html with Wireshark in the 
middle:
With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY from 
the server.
With 9.0.65 - No GOAWAY was sent, and the server and client kept 
ACKing each other.


Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.

Test was conducted with:
Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23)
Chrome Version 109.0.5414.120
JDK 17.0.6+10
Windows 11


Thanks for the reproduction details. I'll take a look now.


A quick workaround is to configure useAsyncIO="false" on the Connector.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.65 suspected memory leak

2023-02-09 Thread Mark Thomas


On 04/02/2023 22:06, Chen Levy wrote:


Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate open 
connections:
I took a fresh Tomcat, unzipped and modified server.xml with only the following:
1. Changed port 8080 to port 80
2. Changed port 8443 to port 443
3. Uncommented the nio connector and added the snippet

 
 
 

I used Chrome to call the default index.html with Wireshark in the middle:
With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY from the 
server.
With 9.0.65 - No GOAWAY was sent, and the server and client kept ACKing each 
other.

Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.

Test was conducted with:
Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23)
Chrome Version 109.0.5414.120
JDK 17.0.6+10
Windows 11


Thanks for the reproduction details. I'll take a look now.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 9.0.65 suspected memory leak

2023-02-04 Thread Chen Levy


> -Original Message-
> From: Mark Thomas 
> Sent: Monday, September 19, 2022 13:02
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.65 suspected memory leak
> 
> On 15/09/2022 14:11, Chen Levy wrote:
> > Hello Experts
> >
> > We’ve recently upgraded some of our production servers to Tomcat
> > 9.0.65; every upgraded server crashed with java.lang.OutOfMemoryError
> > within an hour or so under load.
> >
> > The exact same setup (same application, Linux kernel, Java version
> > etc.) with Tomcat 9.0.63 does not exhibit this issue.
> >
> > A heap-dump through MAT gave the following leak suspect (leak report
> > attached):
> >
> > “
> >
> > 14,364 instances of
> > "org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper", loaded by
> > "java.net.URLClassLoader @ 0x6be257090" occupy 4,489,221,944 (91.95%)
> bytes.
> >
> > These instances are referenced from one instance of
> > "java.util.concurrent.ConcurrentHashMap$Node[]", loaded by " > class loader>", which occupies 590,736 (0.01%) bytes.
> >
> > Keywords
> >
> >      org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper
> >
> >      java.net.URLClassLoader @ 0x6be257090
> >
> >      java.util.concurrent.ConcurrentHashMap$Node[]
> >
> > “
> >
> > Please let me know if I should provide additional information
> 
> That looks like 14k current connections which isn't unreasonable for a
> Tomcat instance under load.
> 
> There are connector related changes between 9.0.63 and 9.0.65 but nothing
> that is obviously related to the issue you are seeing.
> 
> At this point there isn't enough information to differentiate between:
> - a regression introduced in Tomcat between 9.0.63 and 9.0.65
> - a change in Tomcat between 9.0.63 and 9.0.65 that exposed a bug in the
>deployed web application
> - a change in Tomcat between 9.0.63 and 9.0.65 that triggered an
>increase memory usage sufficient to trigger an OOME in your
>environment
> 
> What we would need to investigate this further is a test case that
> demonstrates a leak. It doesn't have to trigger an OOME - it just has to
> demonstrate the JVM retaining references to objects you'd expect to have
> been eligible for GC. If you can reduce it to a single request even better.
> 
> Mark


Mark, I believe a change in Tomcat 9.0.65 causes it to accumulate open 
connections:
I took a fresh Tomcat, unzipped and modified server.xml with only the following:
1. Changed port 8080 to port 80
2. Changed port 8443 to port 443
3. Uncommented the nio connector and added the snippet
   




I used Chrome to call the default index.html with Wireshark in the middle:
With 9.0.63 - 20 seconds after the last data frame, came a GOAWAY from the 
server.
With 9.0.65 - No GOAWAY was sent, and the server and client kept ACKing each 
other.

Tomcat 9.0.71 and 10.1.5 behaved similarly - no GOAWAY was sent.

Test was conducted with:
Wireshark Version 4.0.3 (v4.0.3-0-gc552f74cdc23)
Chrome Version 109.0.5414.120
JDK 17.0.6+10
Windows 11

Chen

Re: Tomcat 9.0.65 suspected memory leak

2022-09-19 Thread Mark Thomas


On 15/09/2022 14:11, Chen Levy wrote:

Hello Experts

We’ve recently upgraded some of our production servers to Tomcat 9.0.65; 
every upgraded server crashed with java.lang.OutOfMemoryError within an 
hour or so under load.


The exact same setup (same application, Linux kernel, Java version etc.) 
with Tomcat 9.0.63 does not exhibit this issue.


A heap-dump through MAT gave the following leak suspect (leak report 
attached):


“

14,364 instances of 
"org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper", loaded by 
"java.net.URLClassLoader @ 0x6be257090" occupy 4,489,221,944 (91.95%) bytes.


These instances are referenced from one instance of 
"java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "class loader>", which occupies 590,736 (0.01%) bytes.


Keywords

     org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper

     java.net.URLClassLoader @ 0x6be257090

     java.util.concurrent.ConcurrentHashMap$Node[]

“

Please let me know if I should provide additional information


That looks like 14k current connections which isn't unreasonable for a 
Tomcat instance under load.


There are connector related changes between 9.0.63 and 9.0.65 but 
nothing that is obviously related to the issue you are seeing.


At this point there isn't enough information to differentiate between:
- a regression introduced in Tomcat between 9.0.63 and 9.0.65
- a change in Tomcat between 9.0.63 and 9.0.65 that exposed a bug in the
  deployed web application
- a change in Tomcat between 9.0.63 and 9.0.65 that triggered an
  increase memory usage sufficient to trigger an OOME in your
  environment

What we would need to investigate this further is a test case that 
demonstrates a leak. It doesn't have to trigger an OOME - it just has to 
demonstrate the JVM retaining references to objects you'd expect to have 
been eligible for GC. If you can reduce it to a single request even better.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 9.0.65 suspected memory leak

2022-09-15 Thread Chen Levy

Hello Experts

We've recently upgraded some of our production servers to Tomcat 9.0.65; every 
upgraded server crashed with java.lang.OutOfMemoryError within an hour or so 
under load.
The exact same setup (same application, Linux kernel, Java version etc.) with 
Tomcat 9.0.63 does not exhibit this issue.

A heap-dump through MAT gave the following leak suspect (leak report attached):

"
14,364 instances of "org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper", 
loaded by "java.net.URLClassLoader @ 0x6be257090" occupy 4,489,221,944 (91.95%) 
bytes.

These instances are referenced from one instance of 
"java.util.concurrent.ConcurrentHashMap$Node[]", loaded by "", which occupies 590,736 (0.01%) bytes.

Keywords

org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper
java.net.URLClassLoader @ 0x6be257090
java.util.concurrent.ConcurrentHashMap$Node[]
"

Please let me know if I should provide additional information

Java: OpenJDK Runtime Environment Temurin-17.0.4+8 (build 17.0.4+8)
Linux: 4.14.281-212.502.amzn2.aarch64








Thanks
Chen
<>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-29 Thread Christopher Schultz


Thomas,

On 3/29/22 02:42, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Mark,


-Ursprüngliche Nachricht-
Von: Mark Eggers 
Gesendet: Montag, 28. März 2022 23:55
An: users@tomcat.apache.org
Betreff: Re: AW: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas:

On 3/28/2022 2:01 PM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Chris,


-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Montag, 28. März 2022 18:48
An: users@tomcat.apache.org
Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:

-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Freitag, 25. März 2022 14:05
An: users@tomcat.apache.org
Betreff: Re: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal
variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH)

wrote:





Is it correct, that every spawned thread must call tl.remove()
to cleanup all

the references to prevent the logged warning (and not only the
main thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static
variable. But

on the other side, that class A is also eligible for garbage
collection after undeployment.

So both, the thread class and the class A are ready to get
garbage collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat
expects clean-up to be completed once contextDestroyed() has
returned for all ServLetContextListeners. If the clean-up is
happening asynchronously

(e.g.

the call is made to stop the threads but doesn't wait until the
threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory
leak but Tomcat can't tell that at the point it checks.

Mark

-
--
-- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the
destroy()

method of a servlet (with load on startup).

At least the debugger shows that servlet-->destroy() is executed
before

the method checkThreadLocalMapForLeaks() runs.

I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for
ThreadLocals, so any threads created by the application or that
library are unrelated to the warning you are seeing.

Any library which saves ThreadLocals from request-processing
threads is going to have this problem if the objects are of types
loaded by the webapp ClassLoader.

There are a few ways to mitigate this, but they are ugly and it
would be better if the library didn't use ThreadLocal storage, or
if it would use vanilla classes from java.* and not its own types.

You say that those objects are eligible for GC after the library
shuts down, but that's not true: anything you stick in ThreadLocal
storage

is being held ...

by the ThreadLocal storage and won't be GC'd. If an object can't be
collected, the java.lang.Class defining it can't be collected, and
therefore the ClassLoader which loaded it (the webapp
ClassLoader) can't be free'd. We call this a "pinned ClassLoader"
and it still contains all of the java.lang.Class instances that the
ClassLoader ever loaded during its lifetime. If you reload
repeatedly, you'll see un-collectable ClassLoader instances piling
up in memory which is
*definitely* a leak.

The good news for you is that Tomcat has noticed the problem and
will, over time, retire and replace each of the affected Threads in
its request- processing thread pool. As those Thread objects are
garbage-collected, the TheradLocal storage for each is also
collected, etc. and *eventually* your leak will be resolved. But it
would be

better not to have one in the first place.


Why not name the library? Why anonymize the object type if it's
org.apache.something?

-chris


Hello Chris,
I didn't want to blame any library  But as you ask for it, I send
more

details.


Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the
Thread-class in the member variable "ThreadLocal.ThreadLocalMap
threadLocals":
https://github.com/AdoptOpenJDK/openjdk-

jdk11/blob/master/src/java.bas

e/share/classes/java/lang/Thread.java

So I thought, when the thread dies, these variables will also be
released and automatically removed from the ThreadLocal variable /
instance (?)

This is correct, but if the ThreadLocal is being stored in the
request- proce

AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-29 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Chris,

> -Ursprüngliche Nachricht-
> Von: Christopher Schultz 
> Gesendet: Montag, 28. März 2022 18:48
> An: users@tomcat.apache.org
> Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
> variable
> 
> Thomas,
> 
> On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz 
> >> Gesendet: Freitag, 25. März 2022 14:05
> >> An: users@tomcat.apache.org
> >> Betreff: Re: AW: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> Thomas,
> >>
> >> On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>
> >>>
> >>>> -Ursprüngliche Nachricht-----
> >>>> Von: Mark Thomas 
> >>>> Gesendet: Donnerstag, 24. März 2022 09:32
> >>>> An: users@tomcat.apache.org
> >>>> Betreff: Re: Question to possible memory leak by Threadlocal
> >>>> variable
> >>>>
> >>>> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>>
> >>>> 
> >>>>
> >>>>> Is it correct, that every spawned thread must call tl.remove() to
> >>>>> cleanup all
> >>>> the references to prevent the logged warning (and not only the main
> >>>> thread)?
> >>>>
> >>>> Yes. Or the threads need to exit.
> >>>>
> >>>>> Second question is: How might it cause a memory leak?
> >>>>> The threads are terminated and hold a reference to this static
> >>>>> variable. But
> >>>> on the other side, that class A is also eligible for garbage
> >>>> collection after undeployment.
> >>>>> So both, the thread class and the class A are ready to get garbage
> >>>>> collected. Maybe I missed something (?)
> >>>>
> >>>> It sounds as if the clean-up is happening too late. Tomcat expects
> >>>> clean-up to be completed once contextDestroyed() has returned for
> >>>> all ServLetContextListeners. If the clean-up is happening
> >>>> asynchronously
> >> (e.g.
> >>>> the call is made to stop the threads but doesn't wait until the
> >>>> threads have
> >>>> stopped) you could see this message.
> >>>>
> >>>> In this case it sounds as if you aren't going to get a memory leak
> >>>> but Tomcat can't tell that at the point it checks.
> >>>>
> >>>> Mark
> >>>>
> >>>> ---
> >>>> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>> Hello Mark,
> >>> thanks for the information.
> >>> The shutdown of the framework is currently placed within the
> >>> destroy()
> >> method of a servlet (with load on startup).
> >>> At least the debugger shows that servlet-->destroy() is executed
> >>> before
> >> the method checkThreadLocalMapForLeaks() runs.
> >>> I will take a look, whether the threads already exited.
> >>
> >> Tomcat only checks its own request-processing threads for
> >> ThreadLocals, so any threads created by the application or that
> >> library are unrelated to the warning you are seeing.
> >>
> >> Any library which saves ThreadLocals from request-processing threads
> >> is going to have this problem if the objects are of types loaded by
> >> the webapp ClassLoader.
> >>
> >> There are a few ways to mitigate this, but they are ugly and it would
> >> be better if the library didn't use ThreadLocal storage, or if it
> >> would use vanilla classes from java.* and not its own types.
> >>
> >> You say that those objects are eligible for GC after the library
> >> shuts down, but that's not true: anything you stick in ThreadLocal storage
> is being held ...
> >> by the ThreadLocal storage and won't be GC'd. If an object can't be
> >> collected, the java.lang.Class defining it can't be collected, and
> >> therefore the ClassLoader which loaded it (the webapp
> >> ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and
> >> it still contains all of the java.lang.Class instances that the
> >>

AW: AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-29 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Mark,

> -Ursprüngliche Nachricht-
> Von: Mark Eggers 
> Gesendet: Montag, 28. März 2022 23:55
> An: users@tomcat.apache.org
> Betreff: Re: AW: AW: AW: Question to possible memory leak by Threadlocal
> variable
> 
> Thomas:
> 
> On 3/28/2022 2:01 PM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello Chris,
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz 
> >> Gesendet: Montag, 28. März 2022 18:48
> >> An: users@tomcat.apache.org
> >> Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> Thomas,
> >>
> >> On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>> -Ursprüngliche Nachricht-
> >>>> Von: Christopher Schultz 
> >>>> Gesendet: Freitag, 25. März 2022 14:05
> >>>> An: users@tomcat.apache.org
> >>>> Betreff: Re: AW: Question to possible memory leak by Threadlocal
> >>>> variable
> >>>>
> >>>> Thomas,
> >>>>
> >>>> On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>>>
> >>>>>
> >>>>>> -Ursprüngliche Nachricht-
> >>>>>> Von: Mark Thomas 
> >>>>>> Gesendet: Donnerstag, 24. März 2022 09:32
> >>>>>> An: users@tomcat.apache.org
> >>>>>> Betreff: Re: Question to possible memory leak by Threadlocal
> >>>>>> variable
> >>>>>>
> >>>>>> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH)
> wrote:
> >>>>>>
> >>>>>> 
> >>>>>>
> >>>>>>> Is it correct, that every spawned thread must call tl.remove()
> >>>>>>> to cleanup all
> >>>>>> the references to prevent the logged warning (and not only the
> >>>>>> main thread)?
> >>>>>>
> >>>>>> Yes. Or the threads need to exit.
> >>>>>>
> >>>>>>> Second question is: How might it cause a memory leak?
> >>>>>>> The threads are terminated and hold a reference to this static
> >>>>>>> variable. But
> >>>>>> on the other side, that class A is also eligible for garbage
> >>>>>> collection after undeployment.
> >>>>>>> So both, the thread class and the class A are ready to get
> >>>>>>> garbage collected. Maybe I missed something (?)
> >>>>>>
> >>>>>> It sounds as if the clean-up is happening too late. Tomcat
> >>>>>> expects clean-up to be completed once contextDestroyed() has
> >>>>>> returned for all ServLetContextListeners. If the clean-up is
> >>>>>> happening asynchronously
> >>>> (e.g.
> >>>>>> the call is made to stop the threads but doesn't wait until the
> >>>>>> threads have
> >>>>>> stopped) you could see this message.
> >>>>>>
> >>>>>> In this case it sounds as if you aren't going to get a memory
> >>>>>> leak but Tomcat can't tell that at the point it checks.
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>> -
> >>>>>> --
> >>>>>> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>>
> >>>>> Hello Mark,
> >>>>> thanks for the information.
> >>>>> The shutdown of the framework is currently placed within the
> >>>>> destroy()
> >>>> method of a servlet (with load on startup).
> >>>>> At least the debugger shows that servlet-->destroy() is executed
> >>>>> before
> >>>> the method checkThreadLocalMapForLeaks() runs.
> >>>>> I will take a look, whether the threads already exited.
> >>>>
> >>>> Tomcat only checks its own request-processing threads for
> >>>> ThreadLocals, so any threads created by the application or that
> >>>> library are unrelated to the warning you are seeing.
> >>>>
> >>>> Any library wh

Re: AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-28 Thread Mark Eggers


Thomas:

On 3/28/2022 2:01 PM, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Chris,


-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Montag, 28. März 2022 18:48
An: users@tomcat.apache.org
Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:

-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Freitag, 25. März 2022 14:05
An: users@tomcat.apache.org
Betreff: Re: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal
variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:




Is it correct, that every spawned thread must call tl.remove() to
cleanup all

the references to prevent the logged warning (and not only the main
thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static
variable. But

on the other side, that class A is also eligible for garbage
collection after undeployment.

So both, the thread class and the class A are ready to get garbage
collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat expects
clean-up to be completed once contextDestroyed() has returned for
all ServLetContextListeners. If the clean-up is happening
asynchronously

(e.g.

the call is made to stop the threads but doesn't wait until the
threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory leak
but Tomcat can't tell that at the point it checks.

Mark

---
-- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the
destroy()

method of a servlet (with load on startup).

At least the debugger shows that servlet-->destroy() is executed
before

the method checkThreadLocalMapForLeaks() runs.

I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for
ThreadLocals, so any threads created by the application or that
library are unrelated to the warning you are seeing.

Any library which saves ThreadLocals from request-processing threads
is going to have this problem if the objects are of types loaded by
the webapp ClassLoader.

There are a few ways to mitigate this, but they are ugly and it would
be better if the library didn't use ThreadLocal storage, or if it
would use vanilla classes from java.* and not its own types.

You say that those objects are eligible for GC after the library
shuts down, but that's not true: anything you stick in ThreadLocal storage

is being held ...

by the ThreadLocal storage and won't be GC'd. If an object can't be
collected, the java.lang.Class defining it can't be collected, and
therefore the ClassLoader which loaded it (the webapp
ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and
it still contains all of the java.lang.Class instances that the
ClassLoader ever loaded during its lifetime. If you reload
repeatedly, you'll see un-collectable ClassLoader instances piling up
in memory which is
*definitely* a leak.

The good news for you is that Tomcat has noticed the problem and
will, over time, retire and replace each of the affected Threads in
its request- processing thread pool. As those Thread objects are
garbage-collected, the TheradLocal storage for each is also
collected, etc. and *eventually* your leak will be resolved. But it would be

better not to have one in the first place.


Why not name the library? Why anonymize the object type if it's
org.apache.something?

-chris


Hello Chris,
I didn't want to blame any library  But as you ask for it, I send more

details.


Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the
Thread-class in the member variable "ThreadLocal.ThreadLocalMap
threadLocals":
https://github.com/AdoptOpenJDK/openjdk-

jdk11/blob/master/src/java.bas

e/share/classes/java/lang/Thread.java

So I thought, when the thread dies, these variables will also be
released and automatically removed from the ThreadLocal variable /
instance (?)

This is correct, but if the ThreadLocal is being stored in the request-
processing thread, then when your web application is redeployed, the
request processing threads outlive that event. Maybe you thought your
application gets a private set of threads for its own use, but that's not the
case: Tomcat pools threads across all applications deployed on the server.

Re: AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-28 Thread Christopher Schultz


Thomas,

On 3/28/22 17:01, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Chris,


-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Montag, 28. März 2022 18:48
An: users@tomcat.apache.org
Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:

-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Freitag, 25. März 2022 14:05
An: users@tomcat.apache.org
Betreff: Re: AW: Question to possible memory leak by Threadlocal
variable

Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal
variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:




Is it correct, that every spawned thread must call tl.remove() to
cleanup all

the references to prevent the logged warning (and not only the main
thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static
variable. But

on the other side, that class A is also eligible for garbage
collection after undeployment.

So both, the thread class and the class A are ready to get garbage
collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat expects
clean-up to be completed once contextDestroyed() has returned for
all ServLetContextListeners. If the clean-up is happening
asynchronously

(e.g.

the call is made to stop the threads but doesn't wait until the
threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory leak
but Tomcat can't tell that at the point it checks.

Mark

---
-- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the
destroy()

method of a servlet (with load on startup).

At least the debugger shows that servlet-->destroy() is executed
before

the method checkThreadLocalMapForLeaks() runs.

I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for
ThreadLocals, so any threads created by the application or that
library are unrelated to the warning you are seeing.

Any library which saves ThreadLocals from request-processing threads
is going to have this problem if the objects are of types loaded by
the webapp ClassLoader.

There are a few ways to mitigate this, but they are ugly and it would
be better if the library didn't use ThreadLocal storage, or if it
would use vanilla classes from java.* and not its own types.

You say that those objects are eligible for GC after the library
shuts down, but that's not true: anything you stick in ThreadLocal storage

is being held ...

by the ThreadLocal storage and won't be GC'd. If an object can't be
collected, the java.lang.Class defining it can't be collected, and
therefore the ClassLoader which loaded it (the webapp
ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and
it still contains all of the java.lang.Class instances that the
ClassLoader ever loaded during its lifetime. If you reload
repeatedly, you'll see un-collectable ClassLoader instances piling up
in memory which is
*definitely* a leak.

The good news for you is that Tomcat has noticed the problem and
will, over time, retire and replace each of the affected Threads in
its request- processing thread pool. As those Thread objects are
garbage-collected, the TheradLocal storage for each is also
collected, etc. and *eventually* your leak will be resolved. But it would be

better not to have one in the first place.


Why not name the library? Why anonymize the object type if it's
org.apache.something?

-chris


Hello Chris,
I didn't want to blame any library  But as you ask for it, I send more

details.


Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the
Thread-class in the member variable "ThreadLocal.ThreadLocalMap
threadLocals":
https://github.com/AdoptOpenJDK/openjdk-

jdk11/blob/master/src/java.bas

e/share/classes/java/lang/Thread.java

So I thought, when the thread dies, these variables will also be
released and automatically removed from the ThreadLocal variable /
instance (?)

This is correct, but if the ThreadLocal is being stored in the request-
processing thread, then when your web application is redeployed, the
request processing threads outlive that event. Maybe you thought your
application gets a private set of threads for its own use, but that's not the
case: Tomcat pools threads across all applications deployed on the server.

AW: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-28 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Chris,

> -Ursprüngliche Nachricht-
> Von: Christopher Schultz 
> Gesendet: Montag, 28. März 2022 18:48
> An: users@tomcat.apache.org
> Betreff: Re: AW: AW: Question to possible memory leak by Threadlocal
> variable
> 
> Thomas,
> 
> On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz 
> >> Gesendet: Freitag, 25. März 2022 14:05
> >> An: users@tomcat.apache.org
> >> Betreff: Re: AW: Question to possible memory leak by Threadlocal
> >> variable
> >>
> >> Thomas,
> >>
> >> On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>
> >>>
> >>>> -Ursprüngliche Nachricht-----
> >>>> Von: Mark Thomas 
> >>>> Gesendet: Donnerstag, 24. März 2022 09:32
> >>>> An: users@tomcat.apache.org
> >>>> Betreff: Re: Question to possible memory leak by Threadlocal
> >>>> variable
> >>>>
> >>>> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>>>
> >>>> 
> >>>>
> >>>>> Is it correct, that every spawned thread must call tl.remove() to
> >>>>> cleanup all
> >>>> the references to prevent the logged warning (and not only the main
> >>>> thread)?
> >>>>
> >>>> Yes. Or the threads need to exit.
> >>>>
> >>>>> Second question is: How might it cause a memory leak?
> >>>>> The threads are terminated and hold a reference to this static
> >>>>> variable. But
> >>>> on the other side, that class A is also eligible for garbage
> >>>> collection after undeployment.
> >>>>> So both, the thread class and the class A are ready to get garbage
> >>>>> collected. Maybe I missed something (?)
> >>>>
> >>>> It sounds as if the clean-up is happening too late. Tomcat expects
> >>>> clean-up to be completed once contextDestroyed() has returned for
> >>>> all ServLetContextListeners. If the clean-up is happening
> >>>> asynchronously
> >> (e.g.
> >>>> the call is made to stop the threads but doesn't wait until the
> >>>> threads have
> >>>> stopped) you could see this message.
> >>>>
> >>>> In this case it sounds as if you aren't going to get a memory leak
> >>>> but Tomcat can't tell that at the point it checks.
> >>>>
> >>>> Mark
> >>>>
> >>>> ---
> >>>> -- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>> Hello Mark,
> >>> thanks for the information.
> >>> The shutdown of the framework is currently placed within the
> >>> destroy()
> >> method of a servlet (with load on startup).
> >>> At least the debugger shows that servlet-->destroy() is executed
> >>> before
> >> the method checkThreadLocalMapForLeaks() runs.
> >>> I will take a look, whether the threads already exited.
> >>
> >> Tomcat only checks its own request-processing threads for
> >> ThreadLocals, so any threads created by the application or that
> >> library are unrelated to the warning you are seeing.
> >>
> >> Any library which saves ThreadLocals from request-processing threads
> >> is going to have this problem if the objects are of types loaded by
> >> the webapp ClassLoader.
> >>
> >> There are a few ways to mitigate this, but they are ugly and it would
> >> be better if the library didn't use ThreadLocal storage, or if it
> >> would use vanilla classes from java.* and not its own types.
> >>
> >> You say that those objects are eligible for GC after the library
> >> shuts down, but that's not true: anything you stick in ThreadLocal storage
> is being held ...
> >> by the ThreadLocal storage and won't be GC'd. If an object can't be
> >> collected, the java.lang.Class defining it can't be collected, and
> >> therefore the ClassLoader which loaded it (the webapp
> >> ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and
> >> it still contains all of the java.lang.Class instances that the
> >>

Re: AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-28 Thread Christopher Schultz


Thomas,

On 3/25/22 16:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:

-Ursprüngliche Nachricht-
Von: Christopher Schultz 
Gesendet: Freitag, 25. März 2022 14:05
An: users@tomcat.apache.org
Betreff: Re: AW: Question to possible memory leak by Threadlocal variable

Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:




Is it correct, that every spawned thread must call tl.remove() to
cleanup all

the references to prevent the logged warning (and not only the main
thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static
variable. But

on the other side, that class A is also eligible for garbage
collection after undeployment.

So both, the thread class and the class A are ready to get garbage
collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat expects
clean-up to be completed once contextDestroyed() has returned for all
ServLetContextListeners. If the clean-up is happening asynchronously

(e.g.

the call is made to stop the threads but doesn't wait until the
threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory leak
but Tomcat can't tell that at the point it checks.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the destroy()

method of a servlet (with load on startup).

At least the debugger shows that servlet-->destroy() is executed before

the method checkThreadLocalMapForLeaks() runs.

I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for ThreadLocals, so
any threads created by the application or that library are unrelated to the
warning you are seeing.

Any library which saves ThreadLocals from request-processing threads is
going to have this problem if the objects are of types loaded by the webapp
ClassLoader.

There are a few ways to mitigate this, but they are ugly and it would be
better if the library didn't use ThreadLocal storage, or if it would use vanilla
classes from java.* and not its own types.

You say that those objects are eligible for GC after the library shuts down,
but that's not true: anything you stick in ThreadLocal storage is being held ...
by the ThreadLocal storage and won't be GC'd. If an object can't be collected,
the java.lang.Class defining it can't be collected, and therefore the
ClassLoader which loaded it (the webapp
ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and it still
contains all of the java.lang.Class instances that the ClassLoader ever loaded
during its lifetime. If you reload repeatedly, you'll see un-collectable
ClassLoader instances piling up in memory which is
*definitely* a leak.

The good news for you is that Tomcat has noticed the problem and will, over
time, retire and replace each of the affected Threads in its request-
processing thread pool. As those Thread objects are garbage-collected, the
TheradLocal storage for each is also collected, etc. and *eventually* your leak
will be resolved. But it would be better not to have one in the first place.

Why not name the library? Why anonymize the object type if it's
org.apache.something?

-chris


Hello Chris,
I didn't want to blame any library  But as you ask for it, I send more details.

Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the
Thread-class in the member variable "ThreadLocal.ThreadLocalMap
threadLocals":
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/lang/Thread.java

So I thought, when the thread dies, these variables will also be
released and automatically removed from the ThreadLocal variable /
instance (?)
This is correct, but if the ThreadLocal is being stored in the 
request-processing thread, then when your web application is redeployed, 
the request processing threads outlive that event. Maybe you thought 
your application gets a private set of threads for its own use, but 
that's not the case: Tomcat pools threads across all applications 
deployed on the server. You can play some games to segregate some 
applications from others, but it's a lot of work for not much gain IMO.


Since the threads outlive the application, you can see the problem, now.


I considered the ThreadLocal class as just the manager of the
thread's member variable "threadLocals&qu

AW: AW: Question to possible memory leak by Threadlocal variable

2022-03-25 Thread Thomas Hoffmann (Speed4Trade GmbH)

> -Ursprüngliche Nachricht-
> Von: Christopher Schultz 
> Gesendet: Freitag, 25. März 2022 14:05
> An: users@tomcat.apache.org
> Betreff: Re: AW: Question to possible memory leak by Threadlocal variable
> 
> Thomas,
> 
> On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Mark Thomas 
> >> Gesendet: Donnerstag, 24. März 2022 09:32
> >> An: users@tomcat.apache.org
> >> Betreff: Re: Question to possible memory leak by Threadlocal variable
> >>
> >> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>
> >> 
> >>
> >>> Is it correct, that every spawned thread must call tl.remove() to
> >>> cleanup all
> >> the references to prevent the logged warning (and not only the main
> >> thread)?
> >>
> >> Yes. Or the threads need to exit.
> >>
> >>> Second question is: How might it cause a memory leak?
> >>> The threads are terminated and hold a reference to this static
> >>> variable. But
> >> on the other side, that class A is also eligible for garbage
> >> collection after undeployment.
> >>> So both, the thread class and the class A are ready to get garbage
> >>> collected. Maybe I missed something (?)
> >>
> >> It sounds as if the clean-up is happening too late. Tomcat expects
> >> clean-up to be completed once contextDestroyed() has returned for all
> >> ServLetContextListeners. If the clean-up is happening asynchronously
> (e.g.
> >> the call is made to stop the threads but doesn't wait until the
> >> threads have
> >> stopped) you could see this message.
> >>
> >> In this case it sounds as if you aren't going to get a memory leak
> >> but Tomcat can't tell that at the point it checks.
> >>
> >> Mark
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> > Hello Mark,
> > thanks for the information.
> > The shutdown of the framework is currently placed within the destroy()
> method of a servlet (with load on startup).
> > At least the debugger shows that servlet-->destroy() is executed before
> the method checkThreadLocalMapForLeaks() runs.
> > I will take a look, whether the threads already exited.
> 
> Tomcat only checks its own request-processing threads for ThreadLocals, so
> any threads created by the application or that library are unrelated to the
> warning you are seeing.
> 
> Any library which saves ThreadLocals from request-processing threads is
> going to have this problem if the objects are of types loaded by the webapp
> ClassLoader.
> 
> There are a few ways to mitigate this, but they are ugly and it would be
> better if the library didn't use ThreadLocal storage, or if it would use 
> vanilla
> classes from java.* and not its own types.
> 
> You say that those objects are eligible for GC after the library shuts down,
> but that's not true: anything you stick in ThreadLocal storage is being held 
> ...
> by the ThreadLocal storage and won't be GC'd. If an object can't be collected,
> the java.lang.Class defining it can't be collected, and therefore the
> ClassLoader which loaded it (the webapp
> ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and it still
> contains all of the java.lang.Class instances that the ClassLoader ever loaded
> during its lifetime. If you reload repeatedly, you'll see un-collectable
> ClassLoader instances piling up in memory which is
> *definitely* a leak.
> 
> The good news for you is that Tomcat has noticed the problem and will, over
> time, retire and replace each of the affected Threads in its request-
> processing thread pool. As those Thread objects are garbage-collected, the
> TheradLocal storage for each is also collected, etc. and *eventually* your 
> leak
> will be resolved. But it would be better not to have one in the first place.
> 
> Why not name the library? Why anonymize the object type if it's
> org.apache.something?
> 
> -chris

Hello Chris,
I didn't want to blame any library  But as you ask for it, I send more details.

Regarding the ThreadLocal thing:
I thought that the threadlocal variables are stored within the Thread-class in 
the member variable "ThreadLocal.ThreadLocalMap threadLocals":
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/lang/Thread.java

Re: AW: Question to possible memory leak by Threadlocal variable

2022-03-25 Thread Christopher Schultz


Thomas,

On 3/24/22 05:49, Thomas Hoffmann (Speed4Trade GmbH) wrote:




-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Donnerstag, 24. März 2022 09:32
An: users@tomcat.apache.org
Betreff: Re: Question to possible memory leak by Threadlocal variable

On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:




Is it correct, that every spawned thread must call tl.remove() to cleanup all

the references to prevent the logged warning (and not only the main
thread)?

Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static variable. But

on the other side, that class A is also eligible for garbage collection after
undeployment.

So both, the thread class and the class A are ready to get garbage
collected. Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat expects clean-up to
be completed once contextDestroyed() has returned for all
ServLetContextListeners. If the clean-up is happening asynchronously (e.g.
the call is made to stop the threads but doesn't wait until the threads have
stopped) you could see this message.

In this case it sounds as if you aren't going to get a memory leak but Tomcat
can't tell that at the point it checks.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the destroy() method 
of a servlet (with load on startup).
At least the debugger shows that servlet-->destroy() is executed before the 
method checkThreadLocalMapForLeaks() runs.
I will take a look, whether the threads already exited.


Tomcat only checks its own request-processing threads for ThreadLocals, 
so any threads created by the application or that library are unrelated 
to the warning you are seeing.


Any library which saves ThreadLocals from request-processing threads is 
going to have this problem if the objects are of types loaded by the 
webapp ClassLoader.


There are a few ways to mitigate this, but they are ugly and it would be 
better if the library didn't use ThreadLocal storage, or if it would use 
vanilla classes from java.* and not its own types.


You say that those objects are eligible for GC after the library shuts 
down, but that's not true: anything you stick in ThreadLocal storage is 
being held ... by the ThreadLocal storage and won't be GC'd. If an 
object can't be collected, the java.lang.Class defining it can't be 
collected, and therefore the ClassLoader which loaded it (the webapp 
ClassLoader) can't be free'd. We call this a "pinned ClassLoader" and it 
still contains all of the java.lang.Class instances that the ClassLoader 
ever loaded during its lifetime. If you reload repeatedly, you'll see 
un-collectable ClassLoader instances piling up in memory which is 
*definitely* a leak.


The good news for you is that Tomcat has noticed the problem and will, 
over time, retire and replace each of the affected Threads in its 
request-processing thread pool. As those Thread objects are 
garbage-collected, the TheradLocal storage for each is also collected, 
etc. and *eventually* your leak will be resolved. But it would be better 
not to have one in the first place.


Why not name the library? Why anonymize the object type if it's 
org.apache.something?


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AW: Question to possible memory leak by Threadlocal variable

2022-03-24 Thread Thomas Hoffmann (Speed4Trade GmbH)



> -Ursprüngliche Nachricht-
> Von: Mark Thomas 
> Gesendet: Donnerstag, 24. März 2022 09:32
> An: users@tomcat.apache.org
> Betreff: Re: Question to possible memory leak by Threadlocal variable
> 
> On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> 
> 
> 
> > Is it correct, that every spawned thread must call tl.remove() to cleanup 
> > all
> the references to prevent the logged warning (and not only the main
> thread)?
> 
> Yes. Or the threads need to exit.
> 
> > Second question is: How might it cause a memory leak?
> > The threads are terminated and hold a reference to this static variable. But
> on the other side, that class A is also eligible for garbage collection after
> undeployment.
> > So both, the thread class and the class A are ready to get garbage
> > collected. Maybe I missed something (?)
> 
> It sounds as if the clean-up is happening too late. Tomcat expects clean-up to
> be completed once contextDestroyed() has returned for all
> ServLetContextListeners. If the clean-up is happening asynchronously (e.g.
> the call is made to stop the threads but doesn't wait until the threads have
> stopped) you could see this message.
> 
> In this case it sounds as if you aren't going to get a memory leak but Tomcat
> can't tell that at the point it checks.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Hello Mark,
thanks for the information.
The shutdown of the framework is currently placed within the destroy() method 
of a servlet (with load on startup).
At least the debugger shows that servlet-->destroy() is executed before the 
method checkThreadLocalMapForLeaks() runs.
I will take a look, whether the threads already exited.

Thanks!
Thomas

Re: Question to possible memory leak by Threadlocal variable

2022-03-24 Thread Mark Thomas


On 24/03/2022 07:57, Thomas Hoffmann (Speed4Trade GmbH) wrote:




Is it correct, that every spawned thread must call tl.remove() to cleanup all 
the references to prevent the logged warning (and not only the main thread)?


Yes. Or the threads need to exit.


Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static variable. But on 
the other side, that class A is also eligible for garbage collection after 
undeployment.
So both, the thread class and the class A are ready to get garbage collected. 
Maybe I missed something (?)


It sounds as if the clean-up is happening too late. Tomcat expects 
clean-up to be completed once contextDestroyed() has returned for all 
ServLetContextListeners. If the clean-up is happening asynchronously 
(e.g. the call is made to stop the threads but doesn't wait until the 
threads have stopped) you could see this message.


In this case it sounds as if you aren't going to get a memory leak but 
Tomcat can't tell that at the point it checks.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Question to possible memory leak by Threadlocal variable

2022-03-24 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello,

we are using a 3rd party lib/framework in our application.
During undeployment we see a warning about a possible memory leak:

org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks 
The web application [ROOT] created a ThreadLocal with key of type 
[java.lang.ThreadLocal.SuppliedThreadLocal] (value 
[java.lang.ThreadLocal$SuppliedThreadLocal@1a14329f]) and a value of type 
[org.apache.xxx] (value [org.apache.xxx]) but failed to remove it when the web 
application was stopped. Threads are going to be renewed over time to try and 
avoid a probable memory leak.

I dug into the sources and found a class A which has a static ThreadLocal 
variable (let's call it tl). During undeployment there is a tl.remove() in the 
shutdown method.
The warning is logged despite that. The framework is spawning several threads 
which might use class A and its threadlocal variable.
Is it correct, that every spawned thread must call tl.remove() to cleanup all 
the references to prevent the logged warning (and not only the main thread)?

Second question is: How might it cause a memory leak?
The threads are terminated and hold a reference to this static variable. But on 
the other side, that class A is also eligible for garbage collection after 
undeployment.
So both, the thread class and the class A are ready to get garbage collected. 
Maybe I missed something (?)

Before filing a bug report to the maintainer of this lib, I would need to 
understand the cause and background of this log entry.

Thanks in advance!
Thomas

RE: [External] Re: Plugging a memory leak - where?

2022-01-10 Thread Scott,Tim

Hi Mark,

Thanks for the prompt, clear response.

> You need to update the version of Tomcat you are running on.
I will advise our deployment team.

It's also worth noting that I need not worry about keeping the Tomcat versions 
up to date in the application source (pom.xml). I will advise our team who 
deals with Dependency Tracker, too.

Thanks,
Tim


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Plugging a memory leak - where?

2022-01-10 Thread Mark Thomas





On 10/01/2022 09:22, Scott,Tim wrote:

Hi all,

We've started to use software from dependency tracker dot com to analyse 
dependencies in our web applications and it highlighted the following:

"The fix for bug 63362 present in Apache Tomcat 10.1.0-M1 to 10.1.0-M5, 10.0.0-M1 to 
10.0.11, 9.0.40 to 9.0.53 and 8.5.60 to 8.5.71 introduced a memory leak. The object 
introduced to collect metrics for HTTP upgrade connections was not released for WebSocket 
connections once the connection was closed. This created a memory leak that, over time, 
could lead to a denial of service via an OutOfMemoryError."

My question would be whether it is sufficient to update the application .war 
file to use, for example, version 9.0.56 of tomcat-servlet-api and related 
libraries or do we also have to deploy with Tomcat 9.0.54+?


No, for several reasons.

1. The code that triggers the memory leak isn't in the API JARs.
2. Java EE API JARs for APIs provided by Tomcat (Servlet, JSP, EL,
   WebSocket and JASPIC) should not be packaged with a web application
   (i.e. they should not be in WEB-INF/lib)
3. Tomcat ignores all Java EE API classes packaged in web applications
   if Tomcat provides that API.


I'm guessing that updating the .war file would remedy the application memory 
leak


Nope. Since there is no application memory leak as a result of this bug.


and updating the deployment version would remedy the memory leak in the Tomcat 
service itself - but I don't like to leave this to guesswork. We've had reports 
of memory leaks causing problems.


You need to update the version of Tomcat you are running on.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Plugging a memory leak - where?

2022-01-10 Thread Scott,Tim

Hi all,

We've started to use software from dependency tracker dot com to analyse 
dependencies in our web applications and it highlighted the following:

"The fix for bug 63362 present in Apache Tomcat 10.1.0-M1 to 10.1.0-M5, 
10.0.0-M1 to 10.0.11, 9.0.40 to 9.0.53 and 8.5.60 to 8.5.71 introduced a memory 
leak. The object introduced to collect metrics for HTTP upgrade connections was 
not released for WebSocket connections once the connection was closed. This 
created a memory leak that, over time, could lead to a denial of service via an 
OutOfMemoryError."

My question would be whether it is sufficient to update the application .war 
file to use, for example, version 9.0.56 of tomcat-servlet-api and related 
libraries or do we also have to deploy with Tomcat 9.0.54+?

I'm guessing that updating the .war file would remedy the application memory 
leak and updating the deployment version would remedy the memory leak in the 
Tomcat service itself - but I don't like to leave this to guesswork. We've had 
reports of memory leaks causing problems.

Thank you,
Tim

--
Tim Scott
OCLC * Senior Software Engineer / Technical Product Manager
CityGate, 8 St. Mary's Gate, Sheffield S1 4LW, UK

cc: IT file

OCLC COVID-19 resources: 
oc.lc/covid19-service-info<https://oc.lc/covid19-service-info>

RE: Potential Memory Leak with StandardManager [EXTERNAL]

2021-10-21 Thread Beard, Shawn

There isn’t a memory leak I'm aware of. At least nothing is reported from what 
I can find. Have you tried setting maxActiveSessions? Its default is -1 which 
means infinite. Also there could be a bug in the code that is just creating new 
sessions for things instead of using the current session.

Shawn   Beard• Sr. Systems Engineer
Middleware Engineering
[cid:image498464.png@B1B8F0D2.A7934045]
3840 109th Street   ,   Urbandale   ,   IA  50322
Phone: +1-515-564-2528
Email:  sbe...@wrberkley.com<mailto:sbe...@wrberkley.com>
Website: https://berkleytechnologyservices.com/
[cid:image170342.jpg@8D7E010B.AF86CC81]
Technology Leadership Unleashing Business Potential

-Original Message-
From: Tim K 
Sent: Tuesday, October 19, 2021 6:35 AM
To: Tomcat Users List 
Subject: Re: Potential Memory Leak with StandardManager [EXTERNAL]

** CAUTION: External message

On Mon, Oct 18, 2021 at 2:51 PM Beard, Shawn 
wrote:

> Update to Tomcat 9.0.54. This could be a known security bug that is
> fixed in this version.
>
> https://urldefense.com/v3/__https://tomcat.apache.org/security-9.html*
> Fixed_in_Apache_Tomcat_9.0.54__;Iw!!Li8W9_Um1Taa!uj_ntV4p2gMeMsLvxyGG7
> 2U9DIPWfNyVeqRi0mD7H1ane3v6ToErlw4gMB9DVcrh$
>

I tried updating to that version but it does not appear to be related; that 
looks to be related to web socket connections.
CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.

Re: Potential Memory Leak with StandardManager [EXTERNAL]

2021-10-19 Thread Tim K

On Mon, Oct 18, 2021 at 2:51 PM Beard, Shawn 
wrote:

> Update to Tomcat 9.0.54. This could be a known security bug that is fixed
> in this version.
>
> https://tomcat.apache.org/security-9.html#Fixed_in_Apache_Tomcat_9.0.54
>

I tried updating to that version but it does not appear to be related; that
looks to be related to web socket connections.

RE: Potential Memory Leak with StandardManager [EXTERNAL]

2021-10-18 Thread Beard, Shawn

Update to Tomcat 9.0.54. This could be a known security bug that is fixed in 
this version.

https://tomcat.apache.org/security-9.html#Fixed_in_Apache_Tomcat_9.0.54


Shawn   Beard• Sr. Systems Engineer
Middleware Engineering
[cid:image624605.png@3C243DDD.ADE52D22]
3840 109th Street   ,   Urbandale   ,   IA  50322
Phone: +1-515-564-2528
Email:  sbe...@wrberkley.com<mailto:sbe...@wrberkley.com>
Website: https://berkleytechnologyservices.com/
[cid:image990961.jpg@8EA0B6A1.B831013E]
Technology Leadership Unleashing Business Potential


-Original Message-
From: Tim K 
Sent: Monday, October 18, 2021 1:07 PM
To: Tomcat Users List 
Subject: Potential Memory Leak with StandardManager [EXTERNAL]

** CAUTION: External message


Running 4 balanced nodes of tomcat 9.0.52 in Linux.

While running with production load, memory usage is slowly growing, it does not 
appear to really drop unless the OS/tomcat is restarted.

I did a load test locally with just login actions, did a heap dump, and MAT 
says:

One instance of org.apache.catalina.session.StandardManager loaded by 
sun.misc.Launcher$AppClassLoader @ 0x9978028 occupies 59,628,760
(57.09%) bytes. The memory is accumulated in one instance of 
java.util.concurrent.ConcurrentHashMap$Node[], loaded by , which occupies 59,615,224 (57.07%) bytes.

My session timeout is set to 15 minutes... I set just a name and userId in the 
session object in our single application running on these servers. Not sure how 
else to troubleshoot and/or resolve this from reaching the heap max.

Are there settings to change in order to get these sessions removed from heap? 
Does GC occur out of the box with tomcat or do I need to define that in order 
for it to happen?

Thanks,
Tim

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.

Potential Memory Leak with StandardManager

2021-10-18 Thread Tim K

Running 4 balanced nodes of tomcat 9.0.52 in Linux.

While running with production load, memory usage is slowly growing, it
does not appear to really drop unless the OS/tomcat is restarted.

I did a load test locally with just login actions, did a heap dump,
and MAT says:

One instance of org.apache.catalina.session.StandardManager loaded by
sun.misc.Launcher$AppClassLoader @ 0x9978028 occupies 59,628,760
(57.09%) bytes. The memory is accumulated in one instance of
java.util.concurrent.ConcurrentHashMap$Node[], loaded by , which occupies 59,615,224 (57.07%) bytes.

My session timeout is set to 15 minutes... I set just a name and
userId in the session object in our single application running on
these servers.  Not sure how else to troubleshoot and/or resolve this
from reaching the heap max.

Are there settings to change in order to get these sessions removed
from heap?  Does GC occur out of the box with tomcat or do I need to
define that in order for it to happen?

Thanks,
Tim

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Possible UpgradeInfo memory leak

2021-09-24 Thread Mark Thomas


On 23/09/2021 09:36, Harri Pesonen wrote:

Hello, while looking at Tomcat 8.5.61 heap dump in VisualVM, in Dominators by 
Retained Size, two biggest ones are:

org.apache.tomcat.util.net.NioEndpoint#1  12 382 781 B (13,7%)
org.apache.coyote.http11.upgrade.UpgradeGroupInfo#1  7 066 212 B (7,8%)

I am wondering about UpgradeGroupInfo, because it has very large array of 
UpgradeInfos:

oname = javax.management.ObjectName#1240 : 
Catalina:Upgrade=websocket,name="https-jsse-nio-10.8.35.86-8443",type=GlobalRequestProcessor
 31 B (0%)  363 B (0%)
upgradeInfos = java.util.ArrayList#10079 : 146 098 elements 
20 B (0%)  7 066 144 B (7,8%)
elementData = java.lang.Object[]#22702 : 160 065 items   640 276 B (0,7%)   
 7 066 124 B (7,8%)
[0] = org.apache.coyote.http11.upgrade.UpgradeInfo#144 B (0%)  44 B 
(0%)

Single UpgradeInfo is very small, but there are 146 098 of them.
org.apache.coyote.http11.upgrade.UpgradeInfo   146 098 (12,8%)

I am not sure what this UpgradeInfo is, it looks like some statistics about 
upgraded connection, in this case from http to websocket.
To me it looks like these UpgradeInfos are not removed when connection is 
closed.
Any comments?


Thanks for the report. We are looking into it.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Possible UpgradeInfo memory leak

2021-09-23 Thread Harri Pesonen

Hello, while looking at Tomcat 8.5.61 heap dump in VisualVM, in Dominators by 
Retained Size, two biggest ones are:

org.apache.tomcat.util.net.NioEndpoint#1  12 382 781 B (13,7%)
org.apache.coyote.http11.upgrade.UpgradeGroupInfo#1  7 066 212 B (7,8%)

I am wondering about UpgradeGroupInfo, because it has very large array of 
UpgradeInfos:

oname = javax.management.ObjectName#1240 : 
Catalina:Upgrade=websocket,name="https-jsse-nio-10.8.35.86-8443",type=GlobalRequestProcessor
 31 B (0%)  363 B (0%)
upgradeInfos = java.util.ArrayList#10079 : 146 098 elements 
20 B (0%)  7 066 144 B (7,8%)
elementData = java.lang.Object[]#22702 : 160 065 items   640 276 B (0,7%)   
 7 066 124 B (7,8%)
[0] = org.apache.coyote.http11.upgrade.UpgradeInfo#144 B (0%)  44 B 
(0%)

Single UpgradeInfo is very small, but there are 146 098 of them.
org.apache.coyote.http11.upgrade.UpgradeInfo   146 098 (12,8%)

I am not sure what this UpgradeInfo is, it looks like some statistics about 
upgraded connection, in this case from http to websocket.
To me it looks like these UpgradeInfos are not removed when connection is 
closed.
Any comments?

Thanks, Harri

Re: AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-06 Thread Mark Thomas


On 06/09/2021 08:47, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Mark,

thank you for taking a look at it and confirming the behaviour.
I will file a bug as suggested.


Thanks, I see it.

I have added my own comment. I'll give others some time to comment and 
aim to fix this for the October release round.


Mark




Thank you and have a good start into the new week!
Thomas

-Ursprüngliche Nachricht-
Von: Mark Thomas 
Gesendet: Montag, 6. September 2021 09:36
An: users@tomcat.apache.org
Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports 
memory leak

Hi Thomas.

Got it.

I think it would be best if you opened a Bugzilla entry for this.

One view is that the JRE should not be doing this - or at least doing it in a 
way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE 
environment. We have raised JRE bugs for issues like this in the past and they 
have been fixed.

The counter view is that this thread is not a one-off and is not short-lived 
(the typical cases for JRE bugs that have been fixed) and Tomcat has already 
acknowledged - with the useContextClassLoader flag - that it needs to be taking 
action to ensure that any threads are created with the expected class loader.

If the consensus is that Tomcat needs to handle this then we'd need to ensure 
that every version of authenticate() also set the class loader if required. 
We'd probably want to refactor the existing handling to make sure we don't try 
setting the class loader twice.

Whatever changes we make we need to be able to reference them back to the 
discussion of why we made them and a Bugzilla issue is how we do that.

Thanks,

Mark


On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Mark,

thanks for your reply.
I already tried that option. But this flag only controls, how the 
InitialDirContext is created.
The worker thread within com.sun.jndi.ldap.Connection is created with that 
corresponding classloader the first time.

The problem is when the worker-thread within the com.sun.jndi.ldap.Connection 
dies and is re-established again.
In that case, the flag useContextClassLoader is not considered any more because 
the InitialDirContext is already instantiated.

The call stack when the InitialDirContext is already instantiated but the 
connection gets re-established looks like:
JNDIRealm.authenticate --> JNDIRealm.getUserBySearch  -->
LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect  -->
LdapClient.getInstance --> Connection.

In this call chain, the flag useContextClassLoader is not used any more as the 
InitialDirContext already exists.
The get() method just provides the existing JNDIConnection without switching 
any classloader.
Now however, the context classloader of the application is used, independent of the 
setting "useContextClassLoader".
Therefore, the second time when the worker thread gets created, it inherits the 
classloader of the application and is reported as leaking during undeployment.

Greetings, Thomas




Von: Mark Thomas 
Gesendet: Sonntag, 5. September 2021 11:55
An: users@tomcat.apache.org
Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads
reports memory leak
  
Thomas,


Try setting:

useContextClassLoader="false"

for the JNDIRealm.

Mark


On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello,

we are using the org.apache.catalina.realm.JNDIRealm for authentication of 
users against our windows AD.
When undeploying the application, we see the following warning in our logs:

WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to 
have started a thread named [Thread-106] but has failed to stop it. This is 
very likely to create a memory leak. Stack trace of thread:
     java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native
Method)
 
java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr

eam.java:115)
 
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja

va:168)
 
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja

va:140)
 
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket

InputRecord.java:448)
 
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet

ePacket(SSLSocketInputRecord.java:68)
 
java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord

(SSLSocketImpl.java:1104)
 
java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S

SLSocketImpl.java:823)
 
java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream

.java:252)
 
java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStrea

m.java:292)
 
java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream

.java:351)
 
java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:8

32)
     java.base@11.0.

AW: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-06 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Mark,

thank you for taking a look at it and confirming the behaviour.
I will file a bug as suggested.

Thank you and have a good start into the new week!
Thomas

-Ursprüngliche Nachricht-
Von: Mark Thomas  
Gesendet: Montag, 6. September 2021 09:36
An: users@tomcat.apache.org
Betreff: Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports 
memory leak

Hi Thomas.

Got it.

I think it would be best if you opened a Bugzilla entry for this.

One view is that the JRE should not be doing this - or at least doing it in a 
way that doesn't run the risk of memory leaks in a Java EE / Jakarta EE 
environment. We have raised JRE bugs for issues like this in the past and they 
have been fixed.

The counter view is that this thread is not a one-off and is not short-lived 
(the typical cases for JRE bugs that have been fixed) and Tomcat has already 
acknowledged - with the useContextClassLoader flag - that it needs to be taking 
action to ensure that any threads are created with the expected class loader.

If the consensus is that Tomcat needs to handle this then we'd need to ensure 
that every version of authenticate() also set the class loader if required. 
We'd probably want to refactor the existing handling to make sure we don't try 
setting the class loader twice.

Whatever changes we make we need to be able to reference them back to the 
discussion of why we made them and a Bugzilla issue is how we do that.

Thanks,

Mark


On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Mark,
> 
> thanks for your reply.
> I already tried that option. But this flag only controls, how the 
> InitialDirContext is created.
> The worker thread within com.sun.jndi.ldap.Connection is created with that 
> corresponding classloader the first time.
> 
> The problem is when the worker-thread within the com.sun.jndi.ldap.Connection 
> dies and is re-established again.
> In that case, the flag useContextClassLoader is not considered any more 
> because the InitialDirContext is already instantiated.
> 
> The call stack when the InitialDirContext is already instantiated but the 
> connection gets re-established looks like:
> JNDIRealm.authenticate --> JNDIRealm.getUserBySearch  --> 
> LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect  --> 
> LdapClient.getInstance --> Connection.
> 
> In this call chain, the flag useContextClassLoader is not used any more as 
> the InitialDirContext already exists.
> The get() method just provides the existing JNDIConnection without switching 
> any classloader.
> Now however, the context classloader of the application is used, independent 
> of the setting "useContextClassLoader".
> Therefore, the second time when the worker thread gets created, it inherits 
> the classloader of the application and is reported as leaking during 
> undeployment.
> 
> Greetings, Thomas
> 
> 
> 
> 
> Von: Mark Thomas 
> Gesendet: Sonntag, 5. September 2021 11:55
> An: users@tomcat.apache.org
> Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads 
> reports memory leak
>  
> Thomas,
> 
> Try setting:
> 
> useContextClassLoader="false"
> 
> for the JNDIRealm.
> 
> Mark
> 
> 
> On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> Hello,
>>
>> we are using the org.apache.catalina.realm.JNDIRealm for authentication of 
>> users against our windows AD.
>> When undeploying the application, we see the following warning in our logs:
>>
>> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp 
>> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to 
>> have started a thread named [Thread-106] but has failed to stop it. This is 
>> very likely to create a memory leak. Stack trace of thread:
>>     java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native 
>> Method)
>>     
>> java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStr
>> eam.java:115)
>>     
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>> va:168)
>>     
>> java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.ja
>> va:140)
>>     
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocket
>> InputRecord.java:448)
>>     
>> java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInComplet
>> ePacket(SSLSocketInputRecord.java:68)
>>     
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord
>> (SSLSocketImpl.java:1104)
>>     
>> java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(S
>> SLSocketImpl.java:823)
>>     
>> java.base@11.0.3/ja

Re: AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-06 Thread Mark Thomas

Hi Thomas.

Got it.

I think it would be best if you opened a Bugzilla entry for this.

One view is that the JRE should not be doing this - or at least doing it
in a way that doesn't run the risk of memory leaks in a Java EE /
Jakarta EE environment. We have raised JRE bugs for issues like this in
the past and they have been fixed.

The counter view is that this thread is not a one-off and is not
short-lived (the typical cases for JRE bugs that have been fixed) and
Tomcat has already acknowledged - with the useContextClassLoader flag -
that it needs to be taking action to ensure that any threads are created
with the expected class loader.

If the consensus is that Tomcat needs to handle this then we'd need to
ensure that every version of authenticate() also set the class loader if
required. We'd probably want to refactor the existing handling to make
sure we don't try setting the class loader twice.

Whatever changes we make we need to be able to reference them back to
the discussion of why we made them and a Bugzilla issue is how we do that.

Thanks,

Mark

On 05/09/2021 22:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello Mark,

thanks for your reply.
I already tried that option. But this flag only controls, how the
InitialDirContext is created.
The worker thread within com.sun.jndi.ldap.Connection is created with that
corresponding classloader the first time.

The problem is when the worker-thread within the com.sun.jndi.ldap.Connection
dies and is re-established again.
In that case, the flag useContextClassLoader is not considered any more because
the InitialDirContext is already instantiated.

The call stack when the InitialDirContext is already instantiated but the
connection gets re-established looks like:
JNDIRealm.authenticate --> JNDIRealm.getUserBySearch --> LdapCtx.dosearch -->
LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance --> Connection.

In this call chain, the flag useContextClassLoader is not used any more as the
InitialDirContext already exists.
The get() method just provides the existing JNDIConnection without switching
any classloader.
Now however, the context classloader of the application is used, independent of the
setting "useContextClassLoader".
Therefore, the second time when the worker thread gets created, it inherits the
classloader of the application and is reported as leaking during undeployment.

Greetings, Thomas

Von: Mark Thomas
Gesendet: Sonntag, 5. September 2021 11:55
An: users@tomcat.apache.org
Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports
memory leak

Thomas,

Try setting:

useContextClassLoader="false"

for the JNDIRealm.

Mark

On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello,

we are using the org.apache.catalina.realm.JNDIRealm for authentication of
users against our windows AD.
When undeploying the application, we see the following warning in our logs:

WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to
have started a thread named [Thread-106] but has failed to stop it. This is
very likely to create a memory leak. Stack trace of thread:
java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
java.base@11.0.3/java.lang.Thread.run(Thread.java:834)

The warning is not always shown but quite often.

Summary of the analysis of the problem:
On tomcat startup, the worker-thread is running under the tomcat classloader.
But when a reconnect happens, the thread is running with the classloader of the
web application and gets thus reported.

AW: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-05 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello Mark,

thanks for your reply.
I already tried that option. But this flag only controls, how the 
InitialDirContext is created.
The worker thread within com.sun.jndi.ldap.Connection is created with that 
corresponding classloader the first time.

The problem is when the worker-thread within the com.sun.jndi.ldap.Connection 
dies and is re-established again.
In that case, the flag useContextClassLoader is not considered any more because 
the InitialDirContext is already instantiated.

The call stack when the InitialDirContext is already instantiated but the 
connection gets re-established looks like:
JNDIRealm.authenticate --> JNDIRealm.getUserBySearch  --> LdapCtx.dosearch --> 
LdapCtx.ensureOpen --> LdapCtx.connect  --> LdapClient.getInstance --> 
Connection.

In this call chain, the flag useContextClassLoader is not used any more as the 
InitialDirContext already exists. 
The get() method just provides the existing JNDIConnection without switching 
any classloader.
Now however, the context classloader of the application is used, independent of 
the setting "useContextClassLoader".
Therefore, the second time when the worker thread gets created, it inherits the 
classloader of the application and is reported as leaking during undeployment.

Greetings, Thomas

Von: Mark Thomas 
Gesendet: Sonntag, 5. September 2021 11:55
An: users@tomcat.apache.org
Betreff: Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports 
memory leak

Thomas,

Try setting:

useContextClassLoader="false"

for the JNDIRealm.

Mark

On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello,
> 
> we are using the org.apache.catalina.realm.JNDIRealm for authentication of 
> users against our windows AD.
> When undeploying the application, we see the following warning in our logs:
> 
> WARNING [Catalina-utility-1] org.apache.catalina.loader.Webapp
> ClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to 
> have started a thread named [Thread-106] but has failed to stop it. This is 
> very likely to create a memory leak. Stack trace of thread:
>   java.base@11.0.3/java.net.SocketInputStream.socketRead0(Native Method)
>   
>java.base@11.0.3/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
>   java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:168)
>   java.base@11.0.3/java.net.SocketInputStream.read(SocketInputStream.java:140)
>   
>java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
>   
>java.base@11.0.3/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
>   
>java.base@11.0.3/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104)
>   
>java.base@11.0.3/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823)
>   
>java.base@11.0.3/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
>   
>java.base@11.0.3/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
>   
>java.base@11.0.3/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
>   java.naming@11.0.3/com.sun.jndi.ldap.Connection.run(Connection.java:832)
>   java.base@11.0.3/java.lang.Thread.run(Thread.java:834)
> 
> The warning is not always shown but quite often.
> 
> Summary of the analysis of the problem:
> On tomcat startup, the worker-thread is running under the tomcat classloader. 
> But when a reconnect happens, the thread is running with the classloader of 
> the web application and gets thus reported.
> 
> The details:
> Digging into the problem via remote debugging showed the reason how this 
> happens:
> During startup, Tomcat is initializing the JNDIRealm. The open-method of 
> JNDIRealm is switching the classloader between bootstrap-CL and 
> tomcat-lib-CL, depending on the attribute "useContextClassLoader".
> Afterwards the context-Object is created (createDirContext). Within this 
> LdapCtx, an LdapClient is used to communicate with the AD-Server.
> This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. 
> This connection opens the reported Worker-Thread.
> This can be seen at  
> https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java
>  around line 243 --> worker = Obj.helper.createThread(this);
> 
> So far, so good.
> 
> Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread 
> dies. At least, the thread is not visible any more. Maybe because of a 
> timeout on the AD-server side or something else happened.
> If a new user accesses the site, the JNDIRealm is authenticating the user.
> This triggers the following chain (path is shortened): 
> JNDIRealm.getUserBySearch --> LdapCtx.dosea

Re: Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-05 Thread Mark Thomas

Thomas,

Try setting:

useContextClassLoader="false"

for the JNDIRealm.

Mark

On 02/09/2021 08:33, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Hello,

we are using the org.apache.catalina.realm.JNDIRealm for authentication of
users against our windows AD.
When undeploying the application, we see the following warning in our logs:

The warning is not always shown but quite often.

The details:
Digging into the problem via remote debugging showed the reason how this
happens:
During startup, Tomcat is initializing the JNDIRealm. The open-method of JNDIRealm is
switching the classloader between bootstrap-CL and tomcat-lib-CL, depending on the
attribute "useContextClassLoader".
Afterwards the context-Object is created (createDirContext). Within this
LdapCtx, an LdapClient is used to communicate with the AD-Server.
This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This
connection opens the reported Worker-Thread.
This can be seen at
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java
around line 243 --> worker = Obj.helper.createThread(this);

So far, so good.

Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread
dies. At least, the thread is not visible any more. Maybe because of a timeout
on the AD-server side or something else happened.
If a new user accesses the site, the JNDIRealm is authenticating the user.
This triggers the following chain (path is shortened): JNDIRealm.getUserBySearch -->
LdapCtx.dosearch --> LdapCtx.ensureOpen --> LdapCtx.connect --> LdapClient.getInstance
--> Connection.
This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this
time, the thread is connected to the classloader of the web-application.
On undeployment, the thread is thus reported to be orphaned.

It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.

As the authentication is conducted within tomcat, before the application is
triggered, I am not sure if the problem can be tackled on application side.

Thanks in advance,
Thomas

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Orphaned thread by JNDIRealm / clearReferencesThreads reports memory leak

2021-09-02 Thread Thomas Hoffmann (Speed4Trade GmbH)

Hello,

we are using the org.apache.catalina.realm.JNDIRealm for authentication of
users against our windows AD.
When undeploying the application, we see the following warning in our logs:

The warning is not always shown but quite often.

The details:
Digging into the problem via remote debugging showed the reason how this
happens:
During startup, Tomcat is initializing the JNDIRealm. The open-method of
JNDIRealm is switching the classloader between bootstrap-CL and tomcat-lib-CL,
depending on the attribute "useContextClassLoader".
Afterwards the context-Object is created (createDirContext). Within this
LdapCtx, an LdapClient is used to communicate with the AD-Server.
This LdapClient uses a com.sun.jndi.ldap.Connection for TCP communication. This
connection opens the reported Worker-Thread.
This can be seen at
https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.naming/share/classes/com/sun/jndi/ldap/Connection.java
around line 243 --> worker = Obj.helper.createThread(this);

So far, so good.

Somehow, the com.sun.jndi.ldap.Connection is sometimes closed and the thread
dies. At least, the thread is not visible any more. Maybe because of a timeout
on the AD-server side or something else happened.
If a new user accesses the site, the JNDIRealm is authenticating the user.
This triggers the following chain (path is shortened):
JNDIRealm.getUserBySearch --> LdapCtx.dosearch --> LdapCtx.ensureOpen -->
LdapCtx.connect --> LdapClient.getInstance --> Connection.
This creates a new com.sun.jndi.ldap.Connection and thus a new thread. But this
time, the thread is connected to the classloader of the web-application.
On undeployment, the thread is thus reported to be orphaned.

It was tested with Tomcat 9.0.52, Windows 10, OpenJDK 11.0.12_7.

As the authentication is conducted within tomcat, before the application is
triggered, I am not sure if the problem can be tackled on application side.

Thanks in advance,
Thomas

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-09 Thread Mark A. Claassen

Thanks for your help, and all the help you provide to people on this list.  
Your replies are always quick, professional, and helpful.

A big part of what makes Tomcat so great is the assistance that can be found on 
this list!

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect 
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
assumes no legal liability or responsibility for the posting. 


-Original Message-
From: Mark Thomas  
Sent: Friday, July 9, 2021 1:31 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak

On 09/07/2021 18:23, Mark A. Claassen wrote:
> Thank you so much!  This is good to know.
> 
> We may not change anything then (until that server gets upgraded).  However, 
> would switching back to HTTP 1.1 reduce the memory footprint in the mean time?

Yes.

The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep some 
information around about the previous few hundred requests on every
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the 
request/response. As of .37/.39 Tomcat replaces these with a much lighter 
object once the request is completed.

> Would the connection be cleaned up when the browser goes away?

Should be.

Mark


> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaas...@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> ---
> Confidentiality Notice: OCIESERVICE
> ---
> The contents of this e-mail message and any attachments are intended solely 
> for the addressee(s) named in this message. This communication is intended to 
> be and to remain confidential. If you are not the intended recipient of this 
> message, or if this message has been addressed to you in error, please 
> immediately alert the sender by reply e-mail and then delete this message and 
> its attachments. Do not deliver, distribute, copy, disclose the contents or 
> take any action in reliance upon the information contained in the 
> communication or any attachments.
> 
> 
> -Original Message-
> From: Mark Thomas 
> Sent: Friday, July 9, 2021 12:59 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> On 09/07/2021 16:21, Mark A. Claassen wrote:
>> Thanks.  I have done more heap analysis and think I have it tracked closer 
>> to the source.
>>
>> --
>> I started looking at the heap a different way.  The random values I looked 
>> at before (of the 80,000) may not have be as representative as I thought.
>>
>> Examining the retained sizes in the heap, I am finding:
>> I have two instances of AbstractProtocol$ConnectionHandler.
>>
>> One of these AbstractProtocol$ConnectionHandler instances has a 
>> ConcurrentHashMap called "connections"
>> This map as 32 elements in its "table".  Most of these are null.  Some of 
>> the ones that are not, are huge.
>> The entirety of the map retains 112MG.
>>
>> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
>> Looking at the "val" variable of a node, there is an 
>> UpgradeProcessorInternal Inside this is a variable called 
>> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I 
>> am looking at now retains 16MG of memory.
>> (Oddly, once I get this far, the retained sizes of its internal 
>> objects don't really add up.)
>>
>> Any ideas on how to work around this?  Or if this is already fixed in a 
>> later version of Tomcat?
> 
> That isn't a leak. That is high memory usage. The objects will be GC'd once 
> the HTTP/2 connection closes.
> 
> The changes in 9.0.37 / 9.0.39 should reduce the memory footprint 
> considerably.
> 
> Mark
> 
> 
>>
>> Thanks,
>>
>> Mark Claassen
>> Senior Software Engineer
>>
>> Donnell Systems, Inc.
>> 130 South Main Street
>> Leighton Plaza Suite 375
>> South Bend, IN  46601
>> E-mail: mailto:mclaas...@ocie.net
>> Voice: (574)232-3784
>> Fax: (574)232-4014
>>
>> Disclaimer:
>> The opinions provided herein do not necessarily state or reflect 
>> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
>> assumes no legal liability or responsibility for the posting

Re: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-09 Thread Mark Thomas


On 09/07/2021 18:23, Mark A. Claassen wrote:

Thank you so much!  This is good to know.

We may not change anything then (until that server gets upgraded).  However, 
would switching back to HTTP 1.1 reduce the memory footprint in the mean time?


Yes.

The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep 
some information around about the previous few hundred requests on every 
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the 
request/response. As of .37/.39 Tomcat replaces these with a much 
lighter object once the request is completed.



Would the connection be cleaned up when the browser goes away?


Should be.

Mark




Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
   
---

Confidentiality Notice: OCIESERVICE
---
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) named in this message. This communication is intended to be 
and to remain confidential. If you are not the intended recipient of this 
message, or if this message has been addressed to you in error, please 
immediately alert the sender by reply e-mail and then delete this message and 
its attachments. Do not deliver, distribute, copy, disclose the contents or 
take any action in reliance upon the information contained in the communication 
or any attachments.


-Original Message-
From: Mark Thomas 
Sent: Friday, July 9, 2021 12:59 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low

On 09/07/2021 16:21, Mark A. Claassen wrote:

Thanks.  I have done more heap analysis and think I have it tracked closer to 
the source.

--
I started looking at the heap a different way.  The random values I looked at 
before (of the 80,000) may not have be as representative as I thought.

Examining the retained sizes in the heap, I am finding:
I have two instances of AbstractProtocol$ConnectionHandler.

One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called 
"connections"
This map as 32 elements in its "table".  Most of these are null.  Some of the 
ones that are not, are huge.
The entirety of the map retains 112MG.

Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
Looking at the "val" variable of a node, there is an
UpgradeProcessorInternal Inside this is a variable called
internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am 
looking at now retains 16MG of memory.
(Oddly, once I get this far, the retained sizes of its internal
objects don't really add up.)

Any ideas on how to work around this?  Or if this is already fixed in a later 
version of Tomcat?


That isn't a leak. That is high memory usage. The objects will be GC'd once the 
HTTP/2 connection closes.

The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.

Mark




Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect those
of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
no legal liability or responsibility for the posting.




-Original Message-
From: Rob Sargent 
Sent: Thursday, July 8, 2021 6:50 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



On 7/8/21 3:17 PM, Mark A. Claassen wrote:

Ok.  That didn’t seem to work.  I will investigate further and try to find a 
way to send that information.

It is not that busy a server, but the memory use increases very quickly.  Doing 
a class_histogram shows MessageBytes growing by the thousands every 30 minutes.

(We have a temporary monitor script in place that does a GC and then
prints a class_histogram every half hour to help us pinpoint what is
happening.)

Thanks,
Mark


Perhaps you've done this already, but

  grep -R 'static HashMap' src

might locate some potential culprits.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




--

RE: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-09 Thread Mark A. Claassen

Thank you so much!  This is good to know.

We may not change anything then (until that server gets upgraded).  However, 
would switching back to HTTP 1.1 reduce the memory footprint in the mean time?

Would the connection be cleaned up when the browser goes away?

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
  
---
Confidentiality Notice: OCIESERVICE
---
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) named in this message. This communication is intended to be 
and to remain confidential. If you are not the intended recipient of this 
message, or if this message has been addressed to you in error, please 
immediately alert the sender by reply e-mail and then delete this message and 
its attachments. Do not deliver, distribute, copy, disclose the contents or 
take any action in reliance upon the information contained in the communication 
or any attachments.


-Original Message-
From: Mark Thomas  
Sent: Friday, July 9, 2021 12:59 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low

On 09/07/2021 16:21, Mark A. Claassen wrote:
> Thanks.  I have done more heap analysis and think I have it tracked closer to 
> the source.
> 
> --
> I started looking at the heap a different way.  The random values I looked at 
> before (of the 80,000) may not have be as representative as I thought.
> 
> Examining the retained sizes in the heap, I am finding:
> I have two instances of AbstractProtocol$ConnectionHandler.
> 
> One of these AbstractProtocol$ConnectionHandler instances has a 
> ConcurrentHashMap called "connections"
> This map as 32 elements in its "table".  Most of these are null.  Some of the 
> ones that are not, are huge.
> The entirety of the map retains 112MG.
> 
> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
> Looking at the "val" variable of a node, there is an 
> UpgradeProcessorInternal Inside this is a variable called 
> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I 
> am looking at now retains 16MG of memory.
> (Oddly, once I get this far, the retained sizes of its internal 
> objects don't really add up.)
> 
> Any ideas on how to work around this?  Or if this is already fixed in a later 
> version of Tomcat?

That isn't a leak. That is high memory usage. The objects will be GC'd once the 
HTTP/2 connection closes.

The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.

Mark


> 
> Thanks,
> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaas...@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
> 
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those 
> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes 
> no legal liability or responsibility for the posting.
> 
> 
> 
> 
> -Original Message-
> From: Rob Sargent 
> Sent: Thursday, July 8, 2021 6:50 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> 
> 
> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>> Ok.  That didn’t seem to work.  I will investigate further and try to find a 
>> way to send that information.
>>
>> It is not that busy a server, but the memory use increases very quickly.  
>> Doing a class_histogram shows MessageBytes growing by the thousands every 30 
>> minutes.
>>
>> (We have a temporary monitor script in place that does a GC and then 
>> prints a class_histogram every half hour to help us pinpoint what is 
>> happening.)
>>
>> Thanks,
>> Mark
>>
> Perhaps you've done this already, but
> 
>  grep -R 'static HashMap' src
> 
> might locate some potential culprits.
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-09 Thread Mark Thomas


On 09/07/2021 16:21, Mark A. Claassen wrote:

Thanks.  I have done more heap analysis and think I have it tracked closer to 
the source.

--
I started looking at the heap a different way.  The random values I looked at 
before (of the 80,000) may not have be as representative as I thought.

Examining the retained sizes in the heap, I am finding:
I have two instances of AbstractProtocol$ConnectionHandler.

One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called 
"connections"
This map as 32 elements in its "table".  Most of these are null.  Some of the 
ones that are not, are huge.
The entirety of the map retains 112MG.

Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
Inside this is a variable called internalHttpUpgradeHandler (of type 
Http2UpgradeHandler)
The one of these I am looking at now retains 16MG of memory.
(Oddly, once I get this far, the retained sizes of its internal objects don't 
really add up.)

Any ideas on how to work around this?  Or if this is already fixed in a later 
version of Tomcat?


That isn't a leak. That is high memory usage. The objects will be GC'd 
once the HTTP/2 connection closes.


The changes in 9.0.37 / 9.0.39 should reduce the memory footprint 
considerably.


Mark




Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.




-Original Message-
From: Rob Sargent 
Sent: Thursday, July 8, 2021 6:50 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



On 7/8/21 3:17 PM, Mark A. Claassen wrote:

Ok.  That didn’t seem to work.  I will investigate further and try to find a 
way to send that information.

It is not that busy a server, but the memory use increases very quickly.  Doing 
a class_histogram shows MessageBytes growing by the thousands every 30 minutes.

(We have a temporary monitor script in place that does a GC and then prints a 
class_histogram every half hour to help us pinpoint what is happening.)

Thanks,
Mark


Perhaps you've done this already, but

 grep -R 'static HashMap' src

might locate some potential culprits.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-09 Thread Mark A. Claassen

Thanks.  I have done more heap analysis and think I have it tracked closer to 
the source.

--
I started looking at the heap a different way.  The random values I looked at 
before (of the 80,000) may not have be as representative as I thought.

Examining the retained sizes in the heap, I am finding:
I have two instances of AbstractProtocol$ConnectionHandler.

One of these AbstractProtocol$ConnectionHandler instances has a 
ConcurrentHashMap called "connections"
This map as 32 elements in its "table".  Most of these are null.  Some of the 
ones that are not, are huge.
The entirety of the map retains 112MG.

Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
Inside this is a variable called internalHttpUpgradeHandler (of type 
Http2UpgradeHandler)
The one of these I am looking at now retains 16MG of memory.
(Oddly, once I get this far, the retained sizes of its internal objects don't 
really add up.)

Any ideas on how to work around this?  Or if this is already fixed in a later 
version of Tomcat?

Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect 
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
assumes no legal liability or responsibility for the posting. 

-Original Message-
From: Rob Sargent  
Sent: Thursday, July 8, 2021 6:50 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low

On 7/8/21 3:17 PM, Mark A. Claassen wrote:
> Ok.  That didn’t seem to work.  I will investigate further and try to find a 
> way to send that information.
>
> It is not that busy a server, but the memory use increases very quickly.  
> Doing a class_histogram shows MessageBytes growing by the thousands every 30 
> minutes.
>
> (We have a temporary monitor script in place that does a GC and then prints a 
> class_histogram every half hour to help us pinpoint what is happening.)
>
> Thanks,
> Mark
>
Perhaps you've done this already, but

grep -R 'static HashMap' src

might locate some potential culprits.

Re: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-08 Thread Rob Sargent




On 7/8/21 3:17 PM, Mark A. Claassen wrote:

Ok.  That didn’t seem to work.  I will investigate further and try to find a 
way to send that information.

It is not that busy a server, but the memory use increases very quickly.  Doing 
a class_histogram shows MessageBytes growing by the thousands every 30 minutes.

(We have a temporary monitor script in place that does a GC and then prints a 
class_histogram every half hour to help us pinpoint what is happening.)

Thanks,
Mark


Perhaps you've done this already, but

   grep -R 'static HashMap' src

might locate some potential culprits.

RE: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-08 Thread Mark A. Claassen

Ok.  That didn’t seem to work.  I will investigate further and try to find a 
way to send that information.

It is not that busy a server, but the memory use increases very quickly.  Doing 
a class_histogram shows MessageBytes growing by the thousands every 30 minutes.

(We have a temporary monitor script in place that does a GC and then prints a 
class_histogram every half hour to help us pinpoint what is happening.)

Thanks,
Mark

From: Mark A. Claassen 
Sent: Thursday, July 8, 2021 5:07 PM
To: Tomcat Users List 
Subject: RE: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low


Thanks for the prompt reply.  The system was not that busy.  Having over 80,000 
seems wrong.  I am going to try to send a picture; I am not sure if this will 
work.  This is from the Netbeans profiler.



[cid:image001.png@01D7741B.B7CC7D90]



-Original Message-
From: Mark Thomas mailto:ma...@apache.org>>
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org<mailto:users@tomcat.apache.org>
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



Memory leak, high memory usage or high GC churn?



The StreamProcessor shouldn't be a GC root. Either something should be 
retaining a reference to it or it should be eligible for GC.



There isn't much in the way of HTTP/2 specific leaks that have been fixed.



For HTTP/2, I'd expect you to see much more impact from the changes around 
9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.



Mark





On 08/07/2021 19:32, Mark A. Claassen wrote:

> Sorry, realized I had a mistyped subject.

>

> -Original Message-

> From: Mark A. Claassen mailto:mclaas...@ocie.net>>

> Sent: Thursday, July 8, 2021 2:30 PM

> To: Tomcat Users List 
> mailto:users@tomcat.apache.org>>

> Subject: Http/s Memory Leak

> Importance: Low

>

> We are using 9.0.12 on a server and noticed a pretty big memory leak.  The 
> change log mentions a some fixed leaks in the releases since 9.0.12.

> I was wondering if this leak was fixed already.

>

> The leak is that there are over 80,000 instances of 
> org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.

> Is this one of the things fixed by setting useAsyncIO="false"?  Is there a 
> downside to using this?

> (We are using Http11AprProtocol and OpenSSL)

>

> Thanks,

> Mark

>

> Reference path of the leak:

> MessageBytes

> MimeHeaderField

> MimeHeaderField[]

> MimeHeaders

> Response

> Stream

> ConcurrentHashMap$Node

> ConcurrentHashMap$Node

> ConcurrentHashMap$Node[]

> Http2UpgradeHandler

> StreamProcessor (Root)

>

> Mark Claassen

> Senior Software Engineer

>

> Donnell Systems, Inc.

> 130 South Main Street

> Leighton Plaza Suite 375

> South Bend, IN  46601

> E-mail: mailto:mclaas...@ocie.net

> Voice: (574)232-3784

> Fax: (574)232-4014

>

> Disclaimer:

> The opinions provided herein do not necessarily state or reflect those of 
> Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal 
> liability or responsibility for the posting.

>

>

> -

> To unsubscribe, e-mail: 
> users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

> For additional commands, e-mail: 
> users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

>

>

> -

> To unsubscribe, e-mail: 
> users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

> For additional commands, e-mail: 
> users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

>





-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

For additional commands, e-mail: 
users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

RE: [Possible Spam] Re: HTTP/2 Memory Leak

2021-07-08 Thread Mark A. Claassen

Thanks for the prompt reply.  The system was not that busy.  Having over 80,000 
seems wrong.  I am going to try to send a picture; I am not sure if this will 
work.  This is from the Netbeans profiler.



[cid:image001.png@01D7741B.B7CC7D90]



-Original Message-
From: Mark Thomas 
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



Memory leak, high memory usage or high GC churn?



The StreamProcessor shouldn't be a GC root. Either something should be 
retaining a reference to it or it should be eligible for GC.



There isn't much in the way of HTTP/2 specific leaks that have been fixed.



For HTTP/2, I'd expect you to see much more impact from the changes around 
9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.



Mark





On 08/07/2021 19:32, Mark A. Claassen wrote:

> Sorry, realized I had a mistyped subject.

>

> -Original Message-

> From: Mark A. Claassen mailto:mclaas...@ocie.net>>

> Sent: Thursday, July 8, 2021 2:30 PM

> To: Tomcat Users List 
> mailto:users@tomcat.apache.org>>

> Subject: Http/s Memory Leak

> Importance: Low

>

> We are using 9.0.12 on a server and noticed a pretty big memory leak.  The 
> change log mentions a some fixed leaks in the releases since 9.0.12.

> I was wondering if this leak was fixed already.

>

> The leak is that there are over 80,000 instances of 
> org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.

> Is this one of the things fixed by setting useAsyncIO="false"?  Is there a 
> downside to using this?

> (We are using Http11AprProtocol and OpenSSL)

>

> Thanks,

> Mark

>

> Reference path of the leak:

> MessageBytes

> MimeHeaderField

> MimeHeaderField[]

> MimeHeaders

> Response

> Stream

> ConcurrentHashMap$Node

> ConcurrentHashMap$Node

> ConcurrentHashMap$Node[]

> Http2UpgradeHandler

> StreamProcessor (Root)

>

> Mark Claassen

> Senior Software Engineer

>

> Donnell Systems, Inc.

> 130 South Main Street

> Leighton Plaza Suite 375

> South Bend, IN  46601

> E-mail: mailto:mclaas...@ocie.net

> Voice: (574)232-3784

> Fax: (574)232-4014

>

> Disclaimer:

> The opinions provided herein do not necessarily state or reflect those of 
> Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal 
> liability or responsibility for the posting.

>

>

> -

> To unsubscribe, e-mail: 
> users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

> For additional commands, e-mail: 
> users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

>

>

> -

> To unsubscribe, e-mail: 
> users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

> For additional commands, e-mail: 
> users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

>





-

To unsubscribe, e-mail: 
users-unsubscr...@tomcat.apache.org<mailto:users-unsubscr...@tomcat.apache.org>

For additional commands, e-mail: 
users-h...@tomcat.apache.org<mailto:users-h...@tomcat.apache.org>

Re: HTTP/2 Memory Leak

2021-07-08 Thread Mark Thomas


Memory leak, high memory usage or high GC churn?

The StreamProcessor shouldn't be a GC root. Either something should be 
retaining a reference to it or it should be eligible for GC.


There isn't much in the way of HTTP/2 specific leaks that have been fixed.

For HTTP/2, I'd expect you to see much more impact from the changes 
around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.


Mark


On 08/07/2021 19:32, Mark A. Claassen wrote:

Sorry, realized I had a mistyped subject.

-Original Message-
From: Mark A. Claassen 
Sent: Thursday, July 8, 2021 2:30 PM
To: Tomcat Users List 
Subject: Http/s Memory Leak
Importance: Low

We are using 9.0.12 on a server and noticed a pretty big memory leak.  The 
change log mentions a some fixed leaks in the releases since 9.0.12.
I was wondering if this leak was fixed already.

The leak is that there are over 80,000 instances of 
org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.
Is this one of the things fixed by setting useAsyncIO="false"?  Is there a 
downside to using this?
(We are using Http11AprProtocol and OpenSSL)

Thanks,
Mark

Reference path of the leak:
MessageBytes
MimeHeaderField
MimeHeaderField[]
MimeHeaders
Response
Stream
ConcurrentHashMap$Node
ConcurrentHashMap$Node
ConcurrentHashMap$Node[]
Http2UpgradeHandler
StreamProcessor (Root)

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect those of 
Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal 
liability or responsibility for the posting.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: HTTP/2 Memory Leak

2021-07-08 Thread Mark A. Claassen

Sorry, realized I had a mistyped subject.

-Original Message-
From: Mark A. Claassen  
Sent: Thursday, July 8, 2021 2:30 PM
To: Tomcat Users List 
Subject: Http/s Memory Leak
Importance: Low

We are using 9.0.12 on a server and noticed a pretty big memory leak.  The 
change log mentions a some fixed leaks in the releases since 9.0.12.  
I was wondering if this leak was fixed already.

The leak is that there are over 80,000 instances of 
org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.
Is this one of the things fixed by setting useAsyncIO="false"?  Is there a 
downside to using this?
(We are using Http11AprProtocol and OpenSSL)

Thanks,
Mark

Reference path of the leak:
MessageBytes 
MimeHeaderField
MimeHeaderField[]
MimeHeaders
Response
Stream
ConcurrentHashMap$Node
ConcurrentHashMap$Node
ConcurrentHashMap$Node[]
Http2UpgradeHandler
StreamProcessor (Root)

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect those of 
Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal 
liability or responsibility for the posting. 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Http/s Memory Leak

2021-07-08 Thread Mark A. Claassen

We are using 9.0.12 on a server and noticed a pretty big memory leak.  The 
change log mentions a some fixed leaks in the releases since 9.0.12.  
I was wondering if this leak was fixed already.

The leak is that there are over 80,000 instances of 
org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.
Is this one of the things fixed by setting useAsyncIO="false"?  Is there a 
downside to using this?
(We are using Http11AprProtocol and OpenSSL)

Thanks,
Mark

Reference path of the leak:
MessageBytes 
MimeHeaderField
MimeHeaderField[]
MimeHeaders
Response
Stream
ConcurrentHashMap$Node
ConcurrentHashMap$Node
ConcurrentHashMap$Node[]
Http2UpgradeHandler
StreamProcessor (Root)

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaas...@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect 
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
assumes no legal liability or responsibility for the posting. 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak with Jersey 2.33

2021-02-01 Thread Mark Eggers


Mark,

On 2/1/2021 1:33 AM, Mark Thomas wrote:

Is the GC root above the only one? I've seen similar behaviour in the
past where weak references appear to be the cause of a leak but closer
inspection uncovers a strong reference.

Mark


Thanks for putting me on the correct track. I'm using log4j2 2.14.0 and 
Jersey REST 2.33 which includes Jackson 2.11.3.


Analyzing the heap dump using Eclipse MAT and excluding weak references 
led me down a fun rabbit hole.


Leak 1:
https://github.com/FasterXML/jackson-core/issues/400

Fix:
in setenv.(bat/sh) add
-Dcom.fasterxml.jackson.core.util.BufferRecyclers.trackReusableBuffers=true

In a servlet context listener add
int released = releaseBuffers(); // with the proper import, of course

Leak 2:
https://issues.apache.org/jira/browse/LOG4J2-578

Although it's marked as fixed, it apparently is not.

Fix (for now):
in setenv(bat/sh) add
-Dlog4j2.disable.jmx=true

With all of that done, undeploying the web application and doing a heap 
dump shows no traces of the offending app (all WebappClassLoaders have 
started=true).


Again, thanks for the tip.

. . . just my two cents.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Memory leak with Jersey 2.33

2021-02-01 Thread Mark Thomas

On 01/02/2021 06:37, Mark Eggers wrote:
> On 1/31/2021 9:39 PM, Mark Eggers wrote:



>> GC roots for cplanapi are:
>>
>> class com.sun.naming.internal.ResourceManager
>> '- propertiesCache java.util.WeakHashMap
>>     '- table java.util.WeakHashMap$Entry[16]
>>    '- java.util.WeakHashMap$Entry
>>   '- referent org.apache.catalina.loader.WebappClassLoader
>>
>> This occurs on both Windows and Linux, and with the above JRE as well
>> as Java 1.8.0_202.
>>
>> . . . just my two cents
>> /mde/
>>
> 
> Please ignore this. Doing two rounds of GC through the manager removed
> the application from memory (as confirmed by a heap dump).
> 
> Now I need to figure out how to force a GC, since we had an OOM
> Metaspace issue with repeated deployments of the application.

Is the GC root above the only one? I've seen similar behaviour in the
past where weak references appear to be the cause of a leak but closer
inspection uncovers a strong reference.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak with Jersey 2.33

2021-01-31 Thread Mark Eggers


On 1/31/2021 9:39 PM, Mark Eggers wrote:

Folks,

This is probably not a Tomcat issue, but any thoughts on how to resolve 
this would be greatly appreciated.


I am running into an apparent ClassLoader leak with the following 
configuration:


Windows 10 Professional (64 bit, latest updates)
OpenJDK 11.0.10
Apache Tomcat 7.0.107
Jersey Rest 2.33

I have a simple application:

ApplicationConfig:

@ApplicationPath("/service")
public class ApplicationConfig extends Application {

     public ApplicationConfig() {
     }

     @Override
     public Set> getClasses(){
     Set> resources = new java.util.HashSet<>();


resources.add(org.mdeggers.cplanapi.resource.CostInfoResource.class);

resources.add(org.mdeggers.cplanapi.resource.InstituteInfoResource.class);

resources.add(org.mdeggers.cplanapi.resource.InstituteTypeInfoResource.class); 




resources.add(org.mdeggers.cplanapi.mapper.NotFoundExceptionMapper.class);

     return resources;
     }
}

One of three resources:

@Path("/v1/inst")
public class InstituteInfoResource {

     public InstituteInfoResource() {
     }

     @GET
     @Path("/info")
     @Produces({MediaType.APPLICATION_JSON})
     public InstituteContainer getNationalList() {
     InstituteInfo instituteinfo = new InstituteInfo();
     return instituteinfo.getNational();
     }

     @GET
     @Path("/info/{ state : [A-Z]{2} }")
     @Produces({MediaType.APPLICATION_JSON})
     public InstituteContainer getStateList(@PathParam("state") String 
state) {

     InstituteInfo instituteinfo = new InstituteInfo();
     return instituteinfo.getState(state);
     }
}

The InstituteInfo class is a DAO that makes requests into a MySQL 
database and returns a container with the information.


This all works, but on undeploying the Tomcat Manager complains that 
there is a leak after unloading the web application.


Doing a heap dump and looking at it with the following OQL in Eclipse 
MAT shows that the application still resides in memory, with started=false.


SELECT wcl.contextName.toString() AS contextName,
    wcl.started AS started,
    wcl.@retainedHeapSize AS retainedSize
FROM org.apache.catalina.loader.WebappClassLoader wcl

contextName   started  retainedSize
/cplanapi false    586,720
/docs true  46,664
/host-manager true  47,288
/manager  true  47,392
/examples true  85,432
   true  46,912

GC roots for cplanapi are:

class com.sun.naming.internal.ResourceManager
'- propertiesCache java.util.WeakHashMap
    '- table java.util.WeakHashMap$Entry[16]
   '- java.util.WeakHashMap$Entry
  '- referent org.apache.catalina.loader.WebappClassLoader

This occurs on both Windows and Linux, and with the above JRE as well as 
Java 1.8.0_202.


. . . just my two cents
/mde/



Please ignore this. Doing two rounds of GC through the manager removed 
the application from memory (as confirmed by a heap dump).


Now I need to figure out how to force a GC, since we had an OOM 
Metaspace issue with repeated deployments of the application.


. . . just my two cents
/mde/



OpenPGP_signature
Description: OpenPGP digital signature

Memory leak with Jersey 2.33

2021-01-31 Thread Mark Eggers


Folks,

This is probably not a Tomcat issue, but any thoughts on how to resolve 
this would be greatly appreciated.


I am running into an apparent ClassLoader leak with the following 
configuration:


Windows 10 Professional (64 bit, latest updates)
OpenJDK 11.0.10
Apache Tomcat 7.0.107
Jersey Rest 2.33

I have a simple application:

ApplicationConfig:

@ApplicationPath("/service")
public class ApplicationConfig extends Application {

public ApplicationConfig() {
}

@Override
public Set> getClasses(){
Set> resources = new java.util.HashSet<>();


resources.add(org.mdeggers.cplanapi.resource.CostInfoResource.class);

resources.add(org.mdeggers.cplanapi.resource.InstituteInfoResource.class);

resources.add(org.mdeggers.cplanapi.resource.InstituteTypeInfoResource.class);


resources.add(org.mdeggers.cplanapi.mapper.NotFoundExceptionMapper.class);

return resources;
}
}

One of three resources:

@Path("/v1/inst")
public class InstituteInfoResource {

public InstituteInfoResource() {
}

@GET
@Path("/info")
@Produces({MediaType.APPLICATION_JSON})
public InstituteContainer getNationalList() {
InstituteInfo instituteinfo = new InstituteInfo();
return instituteinfo.getNational();
}

@GET
@Path("/info/{ state : [A-Z]{2} }")
@Produces({MediaType.APPLICATION_JSON})
public InstituteContainer getStateList(@PathParam("state") String 
state) {

InstituteInfo instituteinfo = new InstituteInfo();
return instituteinfo.getState(state);
}
}

The InstituteInfo class is a DAO that makes requests into a MySQL 
database and returns a container with the information.


This all works, but on undeploying the Tomcat Manager complains that 
there is a leak after unloading the web application.


Doing a heap dump and looking at it with the following OQL in Eclipse 
MAT shows that the application still resides in memory, with started=false.


SELECT wcl.contextName.toString() AS contextName,
   wcl.started AS started,
   wcl.@retainedHeapSize AS retainedSize
FROM org.apache.catalina.loader.WebappClassLoader wcl

contextName   started  retainedSize
/cplanapi false586,720
/docs true  46,664
/host-manager true  47,288
/manager  true  47,392
/examples true  85,432
  true  46,912

GC roots for cplanapi are:

class com.sun.naming.internal.ResourceManager
'- propertiesCache java.util.WeakHashMap
   '- table java.util.WeakHashMap$Entry[16]
  '- java.util.WeakHashMap$Entry
 '- referent org.apache.catalina.loader.WebappClassLoader

This occurs on both Windows and Linux, and with the above JRE as well as 
Java 1.8.0_202.


. . . just my two cents
/mde/



OpenPGP_signature
Description: OpenPGP digital signature

Re: Possible memory leak in AprEndpoint

2021-01-28 Thread roman v

Hi Chris,

The exact version is 9.0.37

Best Regards,
Roman

On Thu, Jan 28, 2021 at 5:17 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Roman,
>
> On 1/28/21 08:00, roman v wrote:
> > I'm using tomcat 9 and in my application, the capacity of used memory
> > by direct buffer pools is increasing over time, after some research I
> > found that in AprEndpoint in method doClose "socketBufferHandler" is
> > not executing "free()" method call, and the link is just set to
> > "EMPTY":> socketBufferHandler = SocketBufferHandler.EMPTY;
> > It looks like because of this allocated native memory is not released.
> >
> > Am I missing something here or is this a bug in tomcat?
>
> What exact version of Tomcat 9 are you running?
>
> -chirs
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Possible memory leak in AprEndpoint

2021-01-28 Thread Christopher Schultz


Roman,

On 1/28/21 08:00, roman v wrote:

I'm using tomcat 9 and in my application, the capacity of used memory
by direct buffer pools is increasing over time, after some research I
found that in AprEndpoint in method doClose "socketBufferHandler" is
not executing "free()" method call, and the link is just set to
"EMPTY":> socketBufferHandler = SocketBufferHandler.EMPTY;
It looks like because of this allocated native memory is not released.

Am I missing something here or is this a bug in tomcat?


What exact version of Tomcat 9 are you running?

-chirs

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Possible memory leak in AprEndpoint

2021-01-28 Thread roman v

Hi!

I'm using tomcat 9 and in my application, the capacity of used memory
by direct buffer pools is increasing over time, after some research I
found that in AprEndpoint in method doClose "socketBufferHandler" is
not executing "free()" method call, and the link is just set to
"EMPTY":> socketBufferHandler = SocketBufferHandler.EMPTY;
It looks like because of this allocated native memory is not released.

Am I missing something here or is this a bug in tomcat?

Best Regards,
Roman

Re: Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky

Okay, I've uploaded the images. Here are the links:

https://i.imgur.com/zAohUmm.png
https://i.imgur.com/SVp7x6R.png
https://i.imgur.com/F97A6b3.png

On Tue, 11 Aug 2020 at 20:21, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Alex,
>
> On 8/11/20 11:47, Alex Maltinsky wrote:
> > Hi Folks
> >
> > We ran into what looks like a memory leak in tomcat 8.5.57 on
> > Ubuntu 18.04 running on Openjdk 11.0.5
> >
> > Our app maintains permanent websocket connections with multiple
> > clients (also written in Java, using the Tyrus websocket client -
> > version 1.13.1). Recently, a few clients began misbehaving. They're
> > opening multiple websocket sessions in parallel which our app
> > doesn't allow. When this happens, the app closes the previous
> > websocket session and starts talking to the client over the new one
> > session exclusively. This issue happened to two of our clients that
> > have a bunch of network connectivity issues which somehow triggered
> > this behavior. However, I'm not here to debug the misbehaving
> > clients, the client issue is being investigated in parallel.
> > However, it appears that this behavior triggered a memory leak in
> > tomcat. Slowly but surely our heap keeps filling up with byte
> > arrays that seem to have come from the misbehaving clients (we know
> > this from the contents of the arrays). Below are the paths to the
> > gc roots for a few of these arrays that contain data from one such
> > client. The heap dump was taken *over an hour after the client was
> > blocked by its IP address using iptables*. So we believe there's no
> > reason for these buffers to still exist in memory. Please note that
> > the path to the GC roots are all inside tomcat, our app is nowhere
> > to be found.
> >
> > We would appreciate any guidance or suggestions.
> >
> > Thanks!
> >
> > image.png
> >
> > image.png
> >
> > image.png
>
> Your image attachments have been removed. Can you post them elsewhere
> + links in your message, or transcribe them into text?
>
> Alternatively, do you have a simple test case that reproduces the proble
> m?
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8y04IACgkQHPApP6U8
> pFhoSBAAjLc+roLwg8Qj7HVaHXGWOyucUqAsQOa7jMucQTQkMfs3PiuIfkMxvRp0
> SdRU0uK81E3gIjkwtjeACyeqZIxjExI2RIIe/A4kwtqcZmHkvE/wgayaWM4ncLlI
> tIrppKyEOb3G5aRwoRlaJ4TZxcm7EnvDKVLC8XTPZeQUtoKBS7zeVQFUZiW72gkM
> UU2RllLITeFsjTZGZfQ6I++XHOgrOMKhvswkxRl0zwSzBZM4Sg44xBUKQPj6b8iQ
> d3t6Z9Cy2ynTP1idI1reqEXHeajHLcrlG3VwDwOzQhhgN0TXL+5HEzomAFin9eaI
> D59tPPEClBmP1+1C3/Bse8XYOTUdDIzA7EoSEgXsaaQbkMmapUx6XUH1l0CGlzhI
> oMUvzkvmMuM4k/MLkD7elsG3YTORLK+ouuwd5gDd6o3xfSy3FXgsAbYxpRMQ/vcs
> zkuJ9trREe9uKCsb/MCtLx4cms3ndTo/Q44cEmpdHVMCeQNKnBeQzn9icEzJU7ib
> cF6fIHoI6ifLw9v1nzO1Lr/v3LiDm2WoptUZSLOcB0+dhaz+gLCId9gz6A0yw2DQ
> VzufR5whkZezS/6Rm40J6r8jLItrXfUALm6eDmulfGDUU6Y9JjQe/oUDdQYArKTE
> lVU7JBCc/4wVrZL6DsfnqDecfprRI01wTjXSJ8IP9s6xega2Czc=
> =R1Cn
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Alex,

On 8/11/20 11:47, Alex Maltinsky wrote:
> Hi Folks
>
> We ran into what looks like a memory leak in tomcat 8.5.57 on
> Ubuntu 18.04 running on Openjdk 11.0.5
>
> Our app maintains permanent websocket connections with multiple
> clients (also written in Java, using the Tyrus websocket client -
> version 1.13.1). Recently, a few clients began misbehaving. They're
> opening multiple websocket sessions in parallel which our app
> doesn't allow. When this happens, the app closes the previous
> websocket session and starts talking to the client over the new one
> session exclusively. This issue happened to two of our clients that
> have a bunch of network connectivity issues which somehow triggered
> this behavior. However, I'm not here to debug the misbehaving
> clients, the client issue is being investigated in parallel.
> However, it appears that this behavior triggered a memory leak in
> tomcat. Slowly but surely our heap keeps filling up with byte
> arrays that seem to have come from the misbehaving clients (we know
> this from the contents of the arrays). Below are the paths to the
> gc roots for a few of these arrays that contain data from one such
> client. The heap dump was taken *over an hour after the client was
> blocked by its IP address using iptables*. So we believe there's no
> reason for these buffers to still exist in memory. Please note that
> the path to the GC roots are all inside tomcat, our app is nowhere
> to be found.
>
> We would appreciate any guidance or suggestions.
>
> Thanks!
>
> image.png
>
> image.png
>
> image.png

Your image attachments have been removed. Can you post them elsewhere
+ links in your message, or transcribe them into text?

Alternatively, do you have a simple test case that reproduces the proble
m?

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8y04IACgkQHPApP6U8
pFhoSBAAjLc+roLwg8Qj7HVaHXGWOyucUqAsQOa7jMucQTQkMfs3PiuIfkMxvRp0
SdRU0uK81E3gIjkwtjeACyeqZIxjExI2RIIe/A4kwtqcZmHkvE/wgayaWM4ncLlI
tIrppKyEOb3G5aRwoRlaJ4TZxcm7EnvDKVLC8XTPZeQUtoKBS7zeVQFUZiW72gkM
UU2RllLITeFsjTZGZfQ6I++XHOgrOMKhvswkxRl0zwSzBZM4Sg44xBUKQPj6b8iQ
d3t6Z9Cy2ynTP1idI1reqEXHeajHLcrlG3VwDwOzQhhgN0TXL+5HEzomAFin9eaI
D59tPPEClBmP1+1C3/Bse8XYOTUdDIzA7EoSEgXsaaQbkMmapUx6XUH1l0CGlzhI
oMUvzkvmMuM4k/MLkD7elsG3YTORLK+ouuwd5gDd6o3xfSy3FXgsAbYxpRMQ/vcs
zkuJ9trREe9uKCsb/MCtLx4cms3ndTo/Q44cEmpdHVMCeQNKnBeQzn9icEzJU7ib
cF6fIHoI6ifLw9v1nzO1Lr/v3LiDm2WoptUZSLOcB0+dhaz+gLCId9gz6A0yw2DQ
VzufR5whkZezS/6Rm40J6r8jLItrXfUALm6eDmulfGDUU6Y9JjQe/oUDdQYArKTE
lVU7JBCc/4wVrZL6DsfnqDecfprRI01wTjXSJ8IP9s6xega2Czc=
=R1Cn
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky

Hi Folks

We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04
running on Openjdk 11.0.5

Our app maintains permanent websocket connections with multiple clients
(also written in Java, using the Tyrus websocket client - version 1.13.1).
Recently, a few clients began misbehaving. They're opening multiple
websocket sessions in parallel which our app doesn't allow. When this
happens, the app closes the previous websocket session and starts talking
to the client over the new one session exclusively. This issue happened to
two of our clients that have a bunch of network connectivity issues which
somehow triggered this behavior. However, I'm not here to debug the
misbehaving clients, the client issue is being investigated in parallel.
However, it appears that this behavior triggered a memory leak in tomcat.
Slowly but surely our heap keeps filling up with byte arrays that seem to
have come from the misbehaving clients (we know this from the contents of
the arrays).
Below are the paths to the gc roots for a few of these arrays that contain
data from one such client. The heap dump was taken *over an hour after the
client was blocked by its IP address using iptables*. So we believe there's
no reason for these buffers to still exist in memory. Please note that the
path to the GC roots are all inside tomcat, our app is nowhere to be found.

We would appreciate any guidance or suggestions.

Thanks!

[image: image.png]

[image: image.png]


[image: image.png]

Fwd: Possible memory leak in Tomcat 8.5.57 Websocket

2020-08-11 Thread Alex Maltinsky

Hi Folks

We ran into what looks like a memory leak in tomcat 8.5.57 on Ubuntu 18.04
running on Openjdk 11.0.5

Our app maintains permanent websocket connections with multiple clients
(also written in Java, using the Tyrus websocket client - version 1.13.1).
Recently, a few clients began misbehaving. They're opening multiple
websocket sessions in parallel which our app doesn't allow. When this
happens, the app closes the previous websocket session and starts talking
to the client over the new one session exclusively. This issue happened to
two of our clients that have a bunch of network connectivity issues which
somehow triggered this behavior. However, I'm not here to debug the
misbehaving clients, the client issue is being investigated in parallel.
However, it appears that this behavior triggered a memory leak in tomcat.
Slowly but surely our heap keeps filling up with byte arrays that seem to
have come from the misbehaving clients (we know this from the contents of
the arrays).
Below are the paths to the gc roots for a few of these arrays that contain
data from one such client. The heap dump was taken *over an hour after the
client was blocked by its IP address using iptables*. So we believe there's
no reason for these buffers to still exist in memory. Please note that the
path to the GC roots are all inside tomcat, our app is nowhere to be found.

We would appreciate any guidance or suggestions.

Thanks!

[image: image.png]

[image: image.png]


[image: image.png]

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Martin Grigorov

On Wed, Jul 22, 2020, 16:58 Ragavendhiran Bhiman (rabhiman)
 wrote:

> Sorry Martin,
>
> My answers are inline.
>

Take a heap dump and analyze it!


> Thanks & Regards,
> Raghav
>
> On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)"
>  wrote:
>
> Hello Martin,
>
> Thanks for your reply
>
>
> https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0
>
> Here is the link I have shared the flame graph.
> Also kindly check my answers in red as well.
>
> Thanks & Regards,
> Raghav
>
>
> On 22/07/20, 2:33 PM, "Martin Grigorov"  wrote:
>
> Hi Ragavendhiran,
>
> On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
>  wrote:
>
> > Hello All,
> >
> >
> >
> > I am seeing the memory leaks from tomcat apache in the following
> SSL path
> > using PKCS11. Attached the flame graph of memory possible memory
> leaks in
> > this area.
> >
> > Please check the attached flame graph of the memory trace. On
> simply a
> > long run the memory keep on allocated in these back traces only
> causing the
> > memory leak, and the polling of the async profiler for more than
> 6hours
> > shows this clearly. Could you please help how to fix this
> problem?
> >
> > (open this svg graph in browser only)
> >
> >
> >
> > Note: If C_DestroyObject is not called because of finalizer
> accumulation
> > is also tested by inducing the gc using the jmap command still
> could see
> > the memory never gone down after the Full GC collection as well.
> Expecting
> > your advice on the same.
> >
>
> With AsyncProfiler '-e alloc' you can see what part of the code is
> responsible for making most of the memory allocations, but it
> doesn't tell
> you whether those objects leak or not. AsyncProfiler helps you to
> identify
> the top allocations and if you manage to reduce them then you will
> reduce
> the GC runs and the time they take.
> To debug memory leaks you need to take a heap dump, e.g. with 'jmap
> -dump:live,format=b,file=heap.bin ' and analyze it. I'd
> recommend you
> to use Eclipse MAT to do that.
>
>  "Yes with async profiler and just leaving the server without
> any sequence of action the memory started growing (RSS grows) when I
> profiled during that sequence there is no other memory allocation happening
> except this one.
> That’s why I am suspecting this flow clearly. Samples are only through
> this flow only".
>
> Also in your flame graph I see that Netty is responsible for
> 49.04% of
> the allocated objects and Tomcat for just 25.32%.
>
>
> >
> > Regards,
> >
> > Raghav
> >
> > Infrastructure engineer,
> >
> > Cisco ISE.
> >
> >
> >
> >
> >
> >
> -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Ragavendhiran Bhiman (rabhiman)

The tomcat version is apache-tomcat-8.5.29
And RedHat Enterprise Linux 7.6

On 22/07/20, 7:28 PM, "Ragavendhiran Bhiman (rabhiman)" 
 wrote:

Hi Chris,

Please see my answers inline.
Also shared the svg graph here.

https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Kindly reply.


Thanks & Regards,
Raghav

On 20/07/20, 11:08 PM, "Christopher Schultz"  
wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Ragavendhiran,

On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
    > Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?
It is being used via the OpenSSL. I am not sure how to identify 
it is via the APR or hardware.

> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.

Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?

Attached again using the dropbox.

    > On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
>  help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
>  accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.

Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.  

 I called the GC via jmap -histo:live but still the weak references 
not cleaned up the RSS never reduced. This means that some where the memory 
being held in native that’s what my suspect. Kindly suggest from your side as 
well.

Thanks.


- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



?B�CB�?�?[��X��ܚX�K??K[XZ[?�?\�\��][��X��ܚX�P??�X�]?�\?X�?K�ܙ�B��܈?Y??]?[ۘ[??��[X[�?�??K[XZ[?�?\�\��Z?[�X�]?�\?X�?K�ܙ�B�

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Ragavendhiran Bhiman (rabhiman)

Hi Chris,

Please see my answers inline.
Also shared the svg graph here.

https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Kindly reply.


Thanks & Regards,
Raghav

On 20/07/20, 11:08 PM, "Christopher Schultz"  
wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Ragavendhiran,

On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
> Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?
It is being used via the OpenSSL. I am not sure how to identify it 
is via the APR or hardware.

> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.

Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?

Attached again using the dropbox.

> On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
>  help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
>  accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.

Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.  

 I called the GC via jmap -histo:live but still the weak references not 
cleaned up the RSS never reduced. This means that some where the memory being 
held in native that’s what my suspect. Kindly suggest from your side as well.

Thanks.


- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Ragavendhiran Bhiman (rabhiman)

Sorry Martin,

My answers are inline.

Thanks & Regards,
Raghav

On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)" 
 wrote:

Hello Martin,

Thanks for your reply

https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Here is the link I have shared the flame graph.
Also kindly check my answers in red as well.

Thanks & Regards,
Raghav

On 22/07/20, 2:33 PM, "Martin Grigorov"  wrote:

Hi Ragavendhiran,

On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
 wrote:

> Hello All,
>
>
>
> I am seeing the memory leaks from tomcat apache in the following SSL 
path
> using PKCS11. Attached the flame graph of memory possible memory 
leaks in
> this area.
>
> Please check the attached flame graph of the memory trace. On simply a
> long run the memory keep on allocated in these back traces only 
causing the
> memory leak, and the polling of the async profiler for more than 
6hours
> shows this clearly. Could you please help how to fix this problem?
>
> (open this svg graph in browser only)
>
>
>
> Note: If C_DestroyObject is not called because of finalizer 
accumulation
> is also tested by inducing the gc using the jmap command still could 
see
> the memory never gone down after the Full GC collection as well. 
Expecting
> your advice on the same.
>

With AsyncProfiler '-e alloc' you can see what part of the code is
responsible for making most of the memory allocations, but it doesn't 
tell
you whether those objects leak or not. AsyncProfiler helps you to 
identify
the top allocations and if you manage to reduce them then you will 
reduce
the GC runs and the time they take.
To debug memory leaks you need to take a heap dump, e.g. with 'jmap
-dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend 
you
to use Eclipse MAT to do that.

 "Yes with async profiler and just leaving the server without any 
sequence of action the memory started growing (RSS grows) when I profiled 
during that sequence there is no other memory allocation happening except this 
one.
That’s why I am suspecting this flow clearly. Samples are only through this 
flow only".

Also in your flame graph I see that Netty is responsible for 49.04% of
the allocated objects and Tomcat for just 25.32%.

>
> Regards,
>
> Raghav
>
> Infrastructure engineer,
>
> Cisco ISE.
>
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Ragavendhiran Bhiman (rabhiman)

Hello Martin,

Thanks for your reply
 
https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0

Here is the link I have shared the flame graph.
Also kindly check my answers in red as well.

Thanks & Regards,
Raghav


On 22/07/20, 2:33 PM, "Martin Grigorov"  wrote:

Hi Ragavendhiran,

On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
 wrote:

> Hello All,
>
>
>
> I am seeing the memory leaks from tomcat apache in the following SSL path
> using PKCS11. Attached the flame graph of memory possible memory leaks in
> this area.
>
> Please check the attached flame graph of the memory trace. On simply a
> long run the memory keep on allocated in these back traces only causing 
the
> memory leak, and the polling of the async profiler for more than 6hours
> shows this clearly. Could you please help how to fix this problem?
>
> (open this svg graph in browser only)
>
>
>
> Note: If C_DestroyObject is not called because of finalizer accumulation
> is also tested by inducing the gc using the jmap command still could see
> the memory never gone down after the Full GC collection as well. Expecting
> your advice on the same.
>

With AsyncProfiler '-e alloc' you can see what part of the code is
responsible for making most of the memory allocations, but it doesn't tell
you whether those objects leak or not. AsyncProfiler helps you to identify
the top allocations and if you manage to reduce them then you will reduce
the GC runs and the time they take.
To debug memory leaks you need to take a heap dump, e.g. with 'jmap
-dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend you
to use Eclipse MAT to do that.

 "Yes with async profiler and just leaving the server without any 
sequence of action the memory started growing (RSS grows) when I profiled 
during that sequence there is no other memory allocation happening except this 
one.
That’s why I am suspecting this flow clearly. Samples are only through this 
flow only".

Also in your flame graph I see that Netty is responsible for 49.04% of
the allocated objects and Tomcat for just 25.32%.


>
> Regards,
>
> Raghav
>
> Infrastructure engineer,
>
> Cisco ISE.
>
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-22 Thread Martin Grigorov

Hi Ragavendhiran,

On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
 wrote:

> Hello All,
>
>
>
> I am seeing the memory leaks from tomcat apache in the following SSL path
> using PKCS11. Attached the flame graph of memory possible memory leaks in
> this area.
>
> Please check the attached flame graph of the memory trace. On simply a
> long run the memory keep on allocated in these back traces only causing the
> memory leak, and the polling of the async profiler for more than 6hours
> shows this clearly. Could you please help how to fix this problem?
>
> (open this svg graph in browser only)
>
>
>
> Note: If C_DestroyObject is not called because of finalizer accumulation
> is also tested by inducing the gc using the jmap command still could see
> the memory never gone down after the Full GC collection as well. Expecting
> your advice on the same.
>

With AsyncProfiler '-e alloc' you can see what part of the code is
responsible for making most of the memory allocations, but it doesn't tell
you whether those objects leak or not. AsyncProfiler helps you to identify
the top allocations and if you manage to reduce them then you will reduce
the GC runs and the time they take.
To debug memory leaks you need to take a heap dump, e.g. with 'jmap
-dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend you
to use Eclipse MAT to do that.

Also in your flame graph I see that Netty is responsible for 49.04% of
the allocated objects and Tomcat for just 25.32%.


>
> Regards,
>
> Raghav
>
> Infrastructure engineer,
>
> Cisco ISE.
>
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-20 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Ragavendhiran,

On 7/18/20 08:53, Ragavendhiran Bhiman (rabhiman) wrote:
> The OS is RHEL => 7.6
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
> Saturday, 18 July 2020 at 6:21 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
>
>
> From: "Ragavendhiran Bhiman (rabhiman)"  Date:
> Saturday, 18 July 2020 at 6:20 PM To: Tomcat Users List
>  Subject: Memory leak in the PKCS11 how to
> fix the problem
>
> Hello All,
>
> I am seeing the memory leaks from tomcat apache in the following
> SSL path using PKCS11.
In what way are you using PKCS#11? Using a hardware crypto engine via
OpenSSL? Are you using the APR connector or are you using OpenSSL via
JSSE?

> Attached the flame graph of memory possible memory leaks in this
> area. Please check the attached flame graph of the memory trace.

Attachments are stripped from messages to this list. Can you post the
graph somewhere and include a link to a reply here?

> On simply a long run the memory keep on allocated in these back
> traces only causing the memory leak, and the polling of the async
> profiler for more than 6hours shows this clearly. Could you please
>  help how to fix this problem? (open this svg graph in browser
> only)> Note: If C_DestroyObject is not called because of finalizer
>  accumulation is also tested by inducing the gc using the jmap
> command still could see the memory never gone down after the Full
> GC collection as well. Expecting your advice on the same.

Are you suggesting that the solution will be to do something explicit
in a class finalizer? Or are you suggesting that some additional
native calls are required to clean-up.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8V1jYACgkQHPApP6U8
pFj3Rw/7B9SxlYNFPZoM6O4GWnwlxI2u+tLrsk2x/6TNOQuFnOIACctTKVe/WIwI
/kF3O7kqwfDEewDhNa6KndUpQgWcvrtbON/QXEAK8SE7HaP0uZv0n5fB6FyRNPPo
uJpnVu5WkJuvFcc9VmluMk9y+SdijhucQtIiK8Y+JEeHmp3Jjg3b45TeJYupDG3K
iAuKqjgWHdRwpVgUNIKxj27PJJBs+qG8yfg7qkVvHrJHzNvZv5rGImcjtwPO6ktQ
U7yWIcJLldF4pkOAWGSiqaCbZsFUCVRK8euejAj5kThZRELKtrJq+QOCy+pIiYHe
rvJpJyG44VLYMqYDMPkDrHVrSJIj88zSJNdfBWXvnC2Ol7dJRTfpau19Js/kYwls
53cELUg7XNOLpH14KnAKk2XMh25xjuZTHLtj2G3p61Frb48+81Ns6X8Hh/oDt/n+
vGUpwU1Crf2HuXg1YiF4L3aw6UnnS3vH/BeeZpj6z8qwFdw9O7ylAcXOndhAiyN0
SxQYmJIXkL8+WXBSTWt7dWZBdYiml/Zl2lkDhRGh+ftCCYZ8UgK5d2+b2HHVQiot
nEs/dAu8rESBkFe0zXB9fgSeviLNwBDkAvenFJ/kZMstT5tzFZiLZW4EKpB8OWTx
fbOGK5KQZB37Cbv+Eq64axWEK3vQHYROfeeAzEz3NgmYE2w74Rg=
=yShM
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-20 Thread Ragavendhiran Bhiman (rabhiman)

HI,

Kindly suggest what could be the issue below.

Thanks & Regards,
Raghav

From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:21 PM
To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem

From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:20 PM
To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem

Hello All,

I am seeing the memory leaks from tomcat apache in the following SSL path 
using PKCS11. Attached the flame graph of memory possible memory leaks in this 
area.
Please check the attached flame graph of the memory trace. On simply a long 
run the memory keep on allocated in these back traces only causing the memory 
leak, and the polling of the async profiler for more than 6hours shows this 
clearly. Could you please help how to fix this problem?
(open this svg graph in browser only)

Note: If C_DestroyObject is not called because of finalizer accumulation is 
also tested by inducing the gc using the jmap command still could see the 
memory never gone down after the Full GC collection as well. Expecting your 
advice on the same.

Regards,
Raghav
Infrastructure engineer,
Cisco ISE.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-18 Thread Ragavendhiran Bhiman (rabhiman)

Kindly reply for the below mail,
Expecting your feedback so that I can take it forward.

Thanks & Regards,
Raghav

On 18/07/20, 6:25 PM, "Ragavendhiran Bhiman (rabhiman)" 
 wrote:

The OS is RHEL => 7.6


From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:21 PM
To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem



From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:20 PM
    To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem

Hello All,

I am seeing the memory leaks from tomcat apache in the following SSL path 
using PKCS11. Attached the flame graph of memory possible memory leaks in this 
area.
Please check the attached flame graph of the memory trace. On simply a long 
run the memory keep on allocated in these back traces only causing the memory 
leak, and the polling of the async profiler for more than 6hours shows this 
clearly. Could you please help how to fix this problem?
(open this svg graph in browser only)

Note: If C_DestroyObject is not called because of finalizer accumulation is 
also tested by inducing the gc using the jmap command still could see the 
memory never gone down after the Full GC collection as well. Expecting your 
advice on the same.

Regards,
Raghav
Infrastructure engineer,
Cisco ISE.

Re: Memory leak in the PKCS11 how to fix the problem

2020-07-18 Thread Ragavendhiran Bhiman (rabhiman)

The OS is RHEL => 7.6

From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:21 PM
To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem

From: "Ragavendhiran Bhiman (rabhiman)" 
Date: Saturday, 18 July 2020 at 6:20 PM
To: Tomcat Users List 
Subject: Memory leak in the PKCS11 how to fix the problem

Hello All,

I am seeing the memory leaks from tomcat apache in the following SSL path using 
PKCS11. Attached the flame graph of memory possible memory leaks in this area.
Please check the attached flame graph of the memory trace. On simply a long run 
the memory keep on allocated in these back traces only causing the memory leak, 
and the polling of the async profiler for more than 6hours shows this clearly. 
Could you please help how to fix this problem?
(open this svg graph in browser only)

Note: If C_DestroyObject is not called because of finalizer accumulation is 
also tested by inducing the gc using the jmap command still could see the 
memory never gone down after the Full GC collection as well. Expecting your 
advice on the same.

Regards,
Raghav
Infrastructure engineer,
Cisco ISE.

Re: suspected memory leak in io.netty.buffer.PoolChunk

2020-04-30 Thread Martin Grigorov

Hi,

On Thu, Apr 30, 2020 at 10:04 AM Ragavendhiran Bhiman (rabhiman)
 wrote:

> Hi All,
>
> When I analysed the heap dump the primary suspect shows that 54.21%
> occupied by the from io.netty.buffer.PoolChunk memory leak.
>

io.netty.* is from Netty project (https://netty.io/index.html)
It seems your application uses it, and probably doesn't close some Netty
resource.
Apache Tomcat itself does NOT use Netty!

I recommend you to use Eclipse MAT (https://www.eclipse.org/mat/) to
analyze the heap dump and see what keeps references to these PoolChunk
instances.

> Kindly help me to proceed further to solve the problem.
> Apache used version is 8.5.29.
>
> Thanks & Regards,
> Raghav
>
> On 29/04/20, 6:53 PM, "Ragavendhiran Bhiman (rabhiman)" <
> rabhi...@cisco.com> wrote:
>
> This question is from 8.5.29 version.
>
> On 29/04/20, 6:52 PM, "Ragavendhiran Bhiman (rabhiman)"
>  wrote:
>
> Hi,
>
> I am seeing a continuous memory leak from
> io.netty.buffer.PoolChunk, Kindly advice how to go ahead and trace the
> problem?
>
> Thanks a lot.
>
> Regards,
> Raghavendran
> +91 8220757651.
>
>
>

Re: suspected memory leak in io.netty.buffer.PoolChunk

2020-04-30 Thread Ragavendhiran Bhiman (rabhiman)

Hi All,

When I analysed the heap dump the primary suspect shows that 54.21% occupied by 
the from io.netty.buffer.PoolChunk memory leak.
Kindly help me to proceed further to solve the problem.
Apache used version is 8.5.29.

Thanks & Regards,
Raghav

On 29/04/20, 6:53 PM, "Ragavendhiran Bhiman (rabhiman)"  
wrote:

This question is from 8.5.29 version.

On 29/04/20, 6:52 PM, "Ragavendhiran Bhiman (rabhiman)" 
 wrote:

Hi,

I am seeing a continuous memory leak from io.netty.buffer.PoolChunk, 
Kindly advice how to go ahead and trace the problem?

Thanks a lot.

Regards,
Raghavendran
+91 8220757651.

1 2 3 4 5 6 7 >

1 - 100 of 606 matches

Mail list logo