Hello,
I'm running Tomcat-8.5 with TLS and I've noticed substantial memory growth
with requests over time, to the point that if I run Tomcat in Docker and
make constant requests to it, Docker will kill the container due to
excessive memory utilization. The problem occurs with standalone Tomcat as
well. Over the course of millions of requests, the memory usage of the
Tomcat process grows continuously, seemingly without bound.
I've done a fair amount of testing on AWS EC2 instances and some local
machines, and here are my observations:
* 'org.apache.tomcat.util.net.openssl.OpenSSLImplementation' seems to
increase memory utilization more quickly and consistently than
'org.apache.tomcat.util.net.jsse.JSSEImplementation'. The
JSSEImplementation doesn't cause the memory to grow in certain setups.
* Limiting the heap size of the JVM does not affect the amount of memory
consumed
* Other than decreasing a small amount (presumably due to garbage
collection), memory utilization stays consistent after the flow of requests
is stopped
My testing consists of:
1. Installing brand new versions of everything
2. Generating a certificate
3. Changing my server.xml to the one shown below
4. Replacing 'webapps/ROOT/index.jsp' with a shorter payload
5. Starting Tomcat
6. Making millions of parallel requests from a different computer and
watching Tomcat's memory utilization grow
Is there anything in my 'server.xml' that would be cause for concern? Are
there any known memory leaks in Tomcat's OpenSSL implementation? What steps
can I take to debug this problem?
Versions:
Tomcat - apache-tomcat-8.5.37
Java - JDK-1.8u191
OpenSSL - openssl-1.0.2q
APR - apr-1.6.5
Tomcat Native - tomcat-native-1.2.19
OS - Amazon Linux release 2 (Karoo)
uname -a - Linux 4.14.77-81.59.amzn2.x86_64 #1 SMP Mon Nov 12 21:32:48 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux
Here is my 'server.xml' file:
<?xml version="1.0" encoding="UTF-8"?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.startup.VersionLoggerListener" />
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener
className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener
className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" />
<Service name="Catalina">
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
port="8443"
SSLEnabled="true" scheme="https" secure="true"
keystoreFile="server.keystore" keystorePass="<REDACTED>"
clientAuth="optional" sslProtocol="TLS"/>
<Engine name="Catalina" defaultHost="localhost">
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log" suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" />
</Host>
</Engine>
</Service>
</Server>
Thanks,
Mason
