On 11/01/2019 17:31, Christopher Schultz wrote: > Mark, > > On 1/11/19 12:23, Mark Thomas wrote: >> On 10/01/2019 19:55, Mark Thomas wrote: > >> <snip/> > >>> I've just tracked down one leak although it is a relatively small >>> one. Next steps are to fix that leak and then find the next one. >>> And then repeat. > >> Found it. > >> The leak impacted NIO and NIO2 when used with OpenSSL. > >> The bug is in Tomcat Native. I have a fix that I am currently >> testing. That fix should be in the next Tomcat Native release. > >> For those interested in the technical details, Tomcat Native >> allocates some Native memory to track various attributes of each >> connection. > >> In APR/Native this memory is allocated from a new APR memory pool >> that is associated with the connection. When the connection closes, >> the pool is destroyed and the memory freed. All is good. > >> In NIO[2]+OpenSSL the per connection memory was allocated from the >> APR memory pool associated with the Connector so the memory was not >> released until the Connector was stopped. The fix is to create a >> pool per connection and use that for the per connection >> allocations. With a fix to do that in place, memory use is now >> broadly static under extended TLS load. > > Nice work. > > I think you can file this under "stuff we knew might be a problem" and > was the reason we were considering reference-counting, etc. to make > sure we didn't leak *connections*. Here, the connection was > cleaned-up, but some of the associated data wasn't being cleaned from > a shared cache. Do I have that right?
Yes. The cache in this case was a memory pool. Think of it as managed malloc. You can destroy pool and clean up all the memory allocated through it without having to call free on each of the individual allocations. It was a tcn_ssl_ctxt_t structure that wasn't being released. It is a collection of 4 pointers so it wasn't too big but one for every connection mounts up over time. > Is there a way, short of bouncing Tomcat, to flush that cache? For > example, bouncing or otherwise re-initializing the connector or > something like that? Yes, but I'm not sure how much practical use it is. You need to unbind the server socket - the necessary clean-up happens AbstractEnpoint.unbind(). That will happen on Connector.stop() or Connector.destroy() depending on bindOnInit. The issue is that unbinding the socket is going to cause connections to be dropped. The end result will be similar to bouncing Tomcat although it should be quicker (no need to reload webapps). Mark --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
