Re: leak on graceful restarts
On Oct 18, 2008, at 3:04 PM, Rainer Jung wrote: Ruediger Pluem schrieb: On 10/18/2008 01:25 AM, Paul Querna wrote: Looking at a problem that seems easy to re-produce using un-patched trunk, 2.2.10 and 2.0.63. Using a graceful restart causes higher memory usage in the parent, which is then passed on to the 'new' children processes. And, at least over here, httpd consistently grows in RSS, without any obvious cause. Seems reproducible on Ubuntu and Darwin, using 2.2.10, 2.0.63 and trunk. Any ideas? Two quick thoughts: 1. Memory fragmentation in the allocator lists (we had this discussion either here or on [EMAIL PROTECTED] a short time ago). 2. At some locations we use a global pool (process-pool) to allocate memory, e.g. mod_ssl and when setting up the listeners. I haven't checked so far if this global pool usage is justified. Using my production configurations on Solaris with 2.2.10 worker I can only reproduce a leak during graceful restart when loading mod_ssl. The memory size does not always increase though, after a couple of restarts it decreases again, but not back to the previous minimum so over all there is a small leak related to restarts. This is weird... I can recreate this under OS X but not under Sol10, and only with mod_ssl in the mix as well. But at least it appears that mod_ssl is the main culprit.
Re: leak on graceful restarts
On 10/19/2008 07:25 PM, Jim Jagielski wrote: On Oct 18, 2008, at 3:04 PM, Rainer Jung wrote: Ruediger Pluem schrieb: On 10/18/2008 01:25 AM, Paul Querna wrote: Looking at a problem that seems easy to re-produce using un-patched trunk, 2.2.10 and 2.0.63. Using a graceful restart causes higher memory usage in the parent, which is then passed on to the 'new' children processes. And, at least over here, httpd consistently grows in RSS, without any obvious cause. Seems reproducible on Ubuntu and Darwin, using 2.2.10, 2.0.63 and trunk. Any ideas? Two quick thoughts: 1. Memory fragmentation in the allocator lists (we had this discussion either here or on [EMAIL PROTECTED] a short time ago). 2. At some locations we use a global pool (process-pool) to allocate memory, e.g. mod_ssl and when setting up the listeners. I haven't checked so far if this global pool usage is justified. Using my production configurations on Solaris with 2.2.10 worker I can only reproduce a leak during graceful restart when loading mod_ssl. The memory size does not always increase though, after a couple of restarts it decreases again, but not back to the previous minimum so over all there is a small leak related to restarts. This is weird... I can recreate this under OS X but not under Sol10, and only with mod_ssl in the mix as well. But at least it appears that mod_ssl is the main culprit. In 2.2.x I guess we leak in ssl_scache_shmcb_init when we create a shared memory segment passing apr_shm_create a global pool. Maybe the actual amount of the leak depends on the platform specific details of the shm implementation. As said this is just a guess. AFAICT this leak does not happen on trunk. Regards Rüdiger
Re: leak on graceful restarts
On 10/18/2008 01:25 AM, Paul Querna wrote: Looking at a problem that seems easy to re-produce using un-patched trunk, 2.2.10 and 2.0.63. Using a graceful restart causes higher memory usage in the parent, which is then passed on to the 'new' children processes. The issue seems to appear in all the Worker, Event and Prefork MPMs. Compile up httpd without apr pool debugging enabled, start it as normal, and then check the memory usage: $ ./apachectl start $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 2672 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3192 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3236 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3280 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start (2672 - 3192 - 3236 - 3280) And, at least over here, httpd consistently grows in RSS, without any obvious cause. Seems reproducible on Ubuntu and Darwin, using 2.2.10, 2.0.63 and trunk. Any ideas? Two quick thoughts: 1. Memory fragmentation in the allocator lists (we had this discussion either here or on [EMAIL PROTECTED] a short time ago). 2. At some locations we use a global pool (process-pool) to allocate memory, e.g. mod_ssl and when setting up the listeners. I haven't checked so far if this global pool usage is justified. Regards Rüdiger
Re: leak on graceful restarts
Ruediger Pluem schrieb: On 10/18/2008 01:25 AM, Paul Querna wrote: Looking at a problem that seems easy to re-produce using un-patched trunk, 2.2.10 and 2.0.63. Using a graceful restart causes higher memory usage in the parent, which is then passed on to the 'new' children processes. The issue seems to appear in all the Worker, Event and Prefork MPMs. Compile up httpd without apr pool debugging enabled, start it as normal, and then check the memory usage: $ ./apachectl start $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 2672 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3192 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3236 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3280 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start (2672 - 3192 - 3236 - 3280) And, at least over here, httpd consistently grows in RSS, without any obvious cause. Seems reproducible on Ubuntu and Darwin, using 2.2.10, 2.0.63 and trunk. Any ideas? Two quick thoughts: 1. Memory fragmentation in the allocator lists (we had this discussion either here or on [EMAIL PROTECTED] a short time ago). 2. At some locations we use a global pool (process-pool) to allocate memory, e.g. mod_ssl and when setting up the listeners. I haven't checked so far if this global pool usage is justified. Using my production configurations on Solaris with 2.2.10 worker I can only reproduce a leak during graceful restart when loading mod_ssl. The memory size does not always increase though, after a couple of restarts it decreases again, but not back to the previous minimum so over all there is a small leak related to restarts. Regards, Rainer
leak on graceful restarts
Looking at a problem that seems easy to re-produce using un-patched trunk, 2.2.10 and 2.0.63. Using a graceful restart causes higher memory usage in the parent, which is then passed on to the 'new' children processes. The issue seems to appear in all the Worker, Event and Prefork MPMs. Compile up httpd without apr pool debugging enabled, start it as normal, and then check the memory usage: $ ./apachectl start $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 2672 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3192 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3236 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start $ ./apachectl graceful $ ps alx | grep 18459 1 1000 18459 1 20 0 152752 3280 674009 Ss ? 0:00 /home/chip/temp/httpd/bin/httpd -k start (2672 - 3192 - 3236 - 3280) And, at least over here, httpd consistently grows in RSS, without any obvious cause. Seems reproducible on Ubuntu and Darwin, using 2.2.10, 2.0.63 and trunk. Any ideas? Thanks, Paul
