Mason, Mark, etc... > -----Original Message----- > From: Mark Thomas <ma...@apache.org> > Sent: Thursday, January 10, 2019 10:45 AM > To: users@tomcat.apache.org > Subject: Re: Tomcat memory growth while using TLS > > On 09/01/2019 14:24, Mark Thomas wrote: > > On 09/01/2019 10:14, Mark Thomas wrote: > > <snip/> > > >> It may be you are seeing the same thing. Or you may have found a > >> memory leak. The next step would be to use a profiler to see where > >> the memory is being used. > > > > Using NIO + OpenSSL with the settings from the commented out > > APR/native TLS connector in server.xml, the heap and non-heap memory > > reported by the profiler reach steady state at fairly low values. > > However, process memory continues to steadily increase. Those > > observations are consistent with a memory leak in native code (but they > are not proof). > > > > Next steps are looking at additional logging to see where the memory > > allocations occur. > > I didn't really get anywhere with that. There was a lot of noise from the JVM > and the performance hit was several orders of magnitude so there was no > obvious built up of leaked memory. > > However, there has been some progress. > > I've tested trunk (9.0.x) with: > - Oracle JDK 1.8.0_192 > - Tomcat Native 1.2.19 > - OpenSSL 1.1.2-dev > - JMeter (20 threads, /index.jsp, no keep-alive) > > and the commented out APT/native TLS connector in the default server.xml. > > APR/native connector - no leak > NIO+JSSE - no leak > NIO+OpenSSL - leak > > Interestingly, the performance of the above was roughly: > NIO+OpenSSL - 5000 req/s > NIO+JSSE - 4000 req/s > APR/native - 400 req/s > > That difference is so large I wonder if I was doing something wrong. > Something to come back to later. > > Given than only NIO+OpenSSL is affected (or at least it is significantly more > affected than the other combinations) I'm going to take another look at the > code focussing on the parts that are specific to that configuration. > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
The first troubleshooting step should be to take a heap dump and analyze it with a tool like Eclipse MAT. I don't know anything about the OpenSSL implementation, but it probably has a session cache. Is there an upper limit on the cache size? The JSSE cache size is unlimited by default, I think, and objects only expire from the cache after 1 day. When a client connects, if it already has a session id from a previous handshake, it will ask the server to resume the session. If it happens to connect to the same server, it will probably get to reuse the existing (cached) session. If not, then a new session is created and cached. If you have a lot of servers behind a load balancer with no stickiness, then the session reuse will not be high. For example, if you have 2 servers, you're only going to get 50% reuse. If you have 100 servers, only 1% reuse! )-: The lower the session reuse rate, the more orphaned sessions there are sitting on the server. Mark, in your test if you were connecting from the same client to the same server over and over, you probably used the same session over and over, so you wouldn't see much session cache growth, if any. I'm not sure of a way to tell the client or server to not reuse sessions for testing purposes. Even round-robining across only 2 servers would be more realistic because the first handshake will result in session 123, then the second (to the other server) would result in session 456 (because the second server doesn't know anything about session 123,) then the third might be to the same server as the first, but by then the client has forgotten about session 123 and tries to resume 456, which doesn't exist on the first server, so you now get session 789 created, and so forth. John
