I've come across an apparent bug that I'm surprised no-one come across
before: Mod_ssl's SSL-session cache handling, both the shmht and shmcb
variants, leaks memory. Not directly (there's no alloc calls in shmcb),
but memory is definitely leaked.
Is this a known bug?
Here are the details on how to reproduce it, and a bit about my guesses
about what's causing this bug:
Try the following: make https requests to an Apache1+mod_ssl server (use
a single-process httpd -X for easier monitoring). The server's memory use
will grow and grow, apparently without stopping, over time. The growth
is relatively slow, but definitely noticible. The growth doesn't happen
when session caching is switched off in the configuration file. The growth
becomes alarmingly quick when the requests are made with client-
authentication (in which case the SSL session objects are much larger).
In fact, in one test, a server with 128MB RAM and configured to use a 5 MB
shared memory (shmht) session cache, was made to exhaust all its memory
and start thrashing in a little less than a minute of stressing it with
client-authenticated https requests; With the same server, and the same
load, but with session caching disabled, the server withstood admiringly
well and an abundance of memory was left.
Monitoring the memory use during the aforementioned test, it was easy to
see how each of Apache's processes (about 50-100 were used during that
high load) was growing and growing, until in about a minute all memory was
finished and intensive paging activity started. This is a good sign of a
memory leak :(
The memory growth cannot be attributed to the shared-memory cache itself
getting gradually filled, because it was only 5 MB large. It had to be
something else. What?
My guess, and I hoped that one of the more experienced people on this
list can help me verify/disprove it, and perhaps even fix this apparent
bug, is that OpenSSL is wasting all that memory. My guess is that though
mod_ssl has "callback functions" implementing an external cache (dbm, shmht,
shmct, and recently distcache) it still keeps its own per-process cache.
This per-process cache is obviously useless for Apache (there's a tiny chance
that a second connection will reach the same process again), but by default
(see SSL_CTX_sess_set_cache_size(3)) as many as 20,000 sessions are kept
in that cache. Even if Apache's configuration is to kill a process after as
little as, say, 1000 connections, this can mean that each process may be
wasting this number of session objects, of sizes 100-1000 bytes. Multiply
100 processes by 1000 sessions by 100-1000 bytes, and you get a total memory
waste of 10MB-100MB...
Even if this guess is wrong, something is definitely leaking memory. Maybe
it's a plain-old-memory-leak inside OpenSSL, but I wasn't able to prove
that (I tried valgrind, which claimed that CRYPTO_malloc was the one that
allocated the leaked memory, but that wasn't much help in finding the
real problem inside OpenSSL. Valgrind also showed me that the growing
memory use was not leaked per se - something was still pointing to it.
But there is no plausible excuse why OpenSSL should be keeping around a
quickly growing state in each process).
If this is something that some mod_ssl developer is planning to look into,
I'd be glad to help in any way I can, and can tell you of more tests and
guesses I've made.
Thanks,
Nadav.
--
Nadav Har'El | Friday, Oct 18 2002, 12 Heshvan 5763
[EMAIL PROTECTED] |-----------------------------------------
Phone: +972-53-245868, ICQ 13349191 |This signature was intentionally left
http://nadav.harel.org.il |boring.
______________________________________________________________________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
