-----Original Message-----
From: Mark Thomas <ma...@apache.org> 
Sent: Thursday, September 26, 2019 15:50
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.24/9.0.26 suspected memory leak

On 26/09/2019 18:22, Chen Levy wrote:
> Hello Experts
> Several of my production servers were recently upgraded from Tomcat 9.0.14 to 
> 9.0.24; immediately after the upgrade the servers started accumulating memory 
> in a steady trend that was not observed before. In addition, CPU utilization 
> that used to hover around 2% not sits at 8%.
> For now the servers are still serving but I suspect they'll become 
> unresponsive in a few hours.
> I loaded a heap dump from one of the servers into MAT and received the 
> following Leak Suspect:
> One instance of "org.apache.coyote.http11.Http11NioProtocol" loaded by 
> "java.net.URLClassLoader @ 0x503f02c40" occupies 9,282,972,608 (96.88%) 
> bytes. The memory is accumulated in one instance of 
> "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class 
> loader>".
> The HashMap referenced in the report appears to be "waitingProcessors" inside 
> AbstractProtocol which contain 262K entries.

OK. Those are asynchronous Servlets that are still in async mode.

While it is possible for an application to deliberately get itself into a state 
like this (infinite async timeouts and don't complete/dispatch the async 
requests) given that it doesn't happen with 9.0.14 but does with 9.0.24 (and 
.26) that suggests a Tomcat bug.

> The same issue was reproduced using v9.0.26 as well
> Please let me know whether I should provide additional information

Can you do a binary search to determine which Tomcat 9.0.x release this problem 
was introduced in?

How easily can you reproduce this? Do you have something approaching a test 
case we could use to repeat the issue?

Meanwhile, I'll take a look at the changelog and see if anything jumps out as a 
possible cause.



> Current setup of the production servers:
> AdoptOpenJDK (build 11.0.3+7)
> Amazon Linux 2
> <Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
>                    maxHttpHeaderSize="16384"
>                    maxThreads="500" minSpareThreads="25"
>                    enableLookups="false" disableUploadTimeout="true"
>                    connectionTimeout="10000"
>                    compression="on"
>                    SSLEnabled="true" scheme="https" secure="true">
>             <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol"
>               keepAliveTimeout="20000"
>                              overheadDataThreadhold="0"/>
>             <SSLHostConfig protocols="+TLSv1.2+TLSv1.3">
>                 <Certificate certificateKeystoreFile="tomcat.keystore"
>                              certificateKeyAlias="tomcat"
>                              certificateKeystorePassword=""
>                              certificateKeystoreType="PKCS12"/>
>             </SSLHostConfig>
> </Connector>
> Thanks
> Chen
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Thanks for the attention Mark, here are some additional information and answers:
* Once the memory was completely consumed, the servers stopped responding with 
CPU stuck at 100%
* I do not employ async servlets in my application
* I cannot do a binary search for a version because of this change: 
 which cause another memory leak and server failure between v9.0.16 and v9.0.21 
and was fixed in v9.0.24 (as far as I know)
* This is easily reproduced with the traffic in my farm and all the servers 
suffer the same. In a development environment it's more tricky; so currently I 
don't have a test case


Reply via email to