FW: Memory leak - Apache2.0.47 and openSSL 0.9.7c
I've been performing some stress tests on the following environment: - Redhat 8.0 - Apache web server 2.0.47 - openSSL 0.9.7c the tests have involved accessing simple web pages many times using an automated load generator. Whilst memory utilization without mod_ssl load was seen to be flat (using gkrellm) - as soon as I loaded mod_ssl and started to use SSL connections memory increases and grows. I've done tests with server authn and with client authn - there would appear to be a memory leak in both cases. In order to obtain more accurate memory utilization metrics due to httpd with mod_ssl I've produced a utility that logged every minute the average RSS over all of the httpd threads I had loaded (1074 in all). In a period of 5 hours I had a leak of 25 Mbytes - per httpd thread A few more details on the test: - 100 virtual clients - total load of about 45 trans/sec - because of a few embedded images this results in about 70-80/sec HTTP GETs - each HTTP GET in general will result in a new SSL connection - in all about 500,000 trans where performed. Please let me know if you would like any other information. I do have output from the load generator and the utility that I can send anyone. John __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
Re: FW: Memory leak - Apache2.0.47 and openSSL 0.9.7c
On Fri, Dec 19, 2003 at 09:19:15AM -, John Hughes wrote: Please let me know if you would like any other information. I do have output from the load generator and the utility that I can send anyone. What type of SSLSessionCache are you using? Do you any any 3rd party modules? Please also note that the current release version is 2.0.48 vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
RE: FW: Memory leak - Apache2.0.47 and openSSL 0.9.7c
I have no 3rd party modules loaded. The testing I did was with and without mod_ssl loaded. Only when mod_ssl was loaded - and SSL was used - did I see a memory leak under load. My SSLSessionCache values are the default and are: SSLSessionCache dbm:logs/ssl_scache SSLSessionCacheTimeout 300 John -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Mads Toftum Sent: 19 December 2003 09:37 To: [EMAIL PROTECTED] Subject: Re: FW: Memory leak - Apache2.0.47 and openSSL 0.9.7c On Fri, Dec 19, 2003 at 09:19:15AM -, John Hughes wrote: Please let me know if you would like any other information. I do have output from the load generator and the utility that I can send anyone. What type of SSLSessionCache are you using? Do you any any 3rd party modules? Please also note that the current release version is 2.0.48 vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED] __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
Re: FW: Memory leak - Apache2.0.47 and openSSL 0.9.7c
On Fri, Dec 19, 2003 at 11:01:57AM -, John Hughes wrote: I have no 3rd party modules loaded. The testing I did was with and without mod_ssl loaded. Only when mod_ssl was loaded - and SSL was used - did I see a memory leak under load. My SSLSessionCache values are the default and are: SSLSessionCache dbm:logs/ssl_scache SSLSessionCacheTimeout300 On linux you really should be using a shared memory session cache - like SSLSessionCache shmcb:logs/ssl_gcache_data(512000) SSLSessionCacheTimeout300 vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
Re: Replacing an SSL certificate
I rebooted the unit eventually, which I guess had the same effect. However I will remember that in future. Many thanks. Huw Jenkins From: Cliff Woolley [EMAIL PROTECTED] Reply-To: [EMAIL PROTECTED] Date: Thu, 18 Dec 2003 14:16:44 -0500 (EST) To: [EMAIL PROTECTED] Subject: Re: Replacing an SSL certificate On Thu, 18 Dec 2003, Huw Jenkins wrote: I've just replaced an SSL certificate (on a Mac OS 10.2 machine) and restarted apache however it's still not recognising the new certificates (still seeing the old one). I've not been asked for the passphrase either. What's the command for getting apache to re read the SSL certificates? What kind of restart did you do? Try stopping and starting again rather than doing a plain restart or a graceful restart. --Cliff __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED] __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
SSL_R_DIGEST_CHECK_FAILED
We have a 0.9.6-based client talking to a 0.9.7a-based mod_ssl server. Communication is fine for initial session negotiation, and for SSL session resumption while the key remains in the cache. However, if the key has expired and we try to pass a new SSL Session ID to the client, the client response is rejected by the server. The error the client is receiving is a handshake error 40 (0x28). The error description generated in the Apache error log is: Library Error: 336117909 error:1408C095:lib(20):func(140):reason(149) lib 20: SSL Library func 140: EC_F_EC_GROUP_GET_FINISHED reason 149: SSL_R_DIGEST_CHECK_FAILED ..and is generated after the server receives the client response to the ServerHello with certificate. The client response consists of a: - ClientKeyExchange - ChangeCipherSpec - EncryptedHandshake For this packet in question. This does *not* happen against a 0.9.6-based mod_ssl of the same version of Apache. Has anyone seen this specific error before in an implementation? (SSL_R_DIGEST_CHECK_FAILED)? Any information would be appreciated. We're frankly scratching our heads as to where this problem is coming from. -- Ken Snider __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
Suggest SSLSessionCacheTimeout and Cache sizes?
Does anyone have any information on shmcb cache sizings? Specifically, how many bytes per request are taken up in shm for each cache entry? I'd like to make sure my shm size is sufficient for the Cache Timeouts I want to use. Secondly, is there any reason why the SSLSessionCacheTimeout can't be arbitrarily large (say, an hour)? And at what size (or number of entries) does the cache size begin to seriously hamper lookups within the cache itself? Thanks for any assistance with the above. -- Ken Snider __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
Re: Suggest SSLSessionCacheTimeout and Cache sizes?
Hi Ken, On December 19, 2003 03:54 pm, Ken Snider wrote: Does anyone have any information on shmcb cache sizings? Specifically, how many bytes per request are taken up in shm for each cache entry? I'd like to make sure my shm size is sufficient for the Cache Timeouts I want to use. The geometry and administrative data in the shared-memory segment create their own nominal overheads, but for the most part the space taken to store a session is largely determined by openssl's asn1 encoding of the corresponding SSL_SESSION object. For a typical ssl session, you'd expect this to be around 120-150 bytes. OTOH if you use client certificate verification, things get a bit more serious - ie. the bulk of the encoded session is a copy of the client-certificate, so this can easily grow to 1Kb or more. As for the organisation of the cache data, I can blab on about that - but won't do so unless dared. Secondly, is there any reason why the SSLSessionCacheTimeout can't be arbitrarily large (say, an hour)? And at what size (or number of entries) does the cache size begin to seriously hamper lookups within the cache itself? This was a problem with the earlier caching code (shmht) and was part of my motivation for designing shmcb (though there were numerous other reasons, bugs and slothful speed being two major ones). Namely, the cache would essentially block when full and fail to add new sessions until an expiry-checker would periodically scan over the cache and delete anything old. This meant you needed to choose cache timeouts carefully to avoid this situation - because if you're under load and the cache fills, you're going to be even *more* under load on account of attempted session resumes resorting to full handshakes more and more often. shmcb uses a different approach to shmht - if the cache fills (or more accurately, if the sub-cache corresponding to a session you're trying to add fills), then sessions are scrolled out of the (sub-)cache until there is enough room for the new one. Sessions are prematurely expired in order of their expected expiry time - that's a complicated phrase because it's not necessarily quite the same thing as the oldest sessions, because you may be using vhosts that use different expiry times for sessions. The consequence of this behaviour is that if you have limited shared memory for the cache and your load is sufficient that sessions generally get prematurely removed before reaching their specified timeout(s), then so be it - the effective timeout implicitly adapts to maintain a full cache and to favour newly added sessions over those that have been there a while. When the load drops again, this effective timeout grows until that point where sessions begin to reach their natural expiry time and are then removed irrespective of cache utilisation. Moreover, the use of the cyclic buffers, sub-caches, and indexing tricks means that the speed of cache operations should be pretty constant irrespective of the cache size or utilisation. Which is all a rather long-winded way of saying yes to your Secondly,... question. With shmcb you can choose the timeout according to security arguments alone, cache size and usage will automatically remove sessions prematurely when necessary to keep things flowing nicely, so the timeout is more of an administrative setting that says if the session is still in the cache after *this* many seconds, I want it explicitly removed to force any future resume attempts to undergo a full handshake anyway. The timeout only dictates what happens when the cache has space to spare. For more information, this old post linked off to 2 or 3 other posts with all the run-down on shmcb and associated junk; http://marc.theaimsgroup.com/?l=apache-modsslm=102286849318705w=2 Or searching for shmcb in the archives can pull up more recent threads too. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ __ Apache Interface to OpenSSL (mod_ssl) www.modssl.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager[EMAIL PROTECTED]
