Re: httpd memory consumption
On 10/06/2017 10:26 AM, Bert Huijben wrote: > > >> -Original Message- >> From: Ruediger Pluem [mailto:rpl...@apache.org] >> Sent: vrijdag 6 oktober 2017 09:47 >> To: Apache HTTP Server Development List >> Subject: httpd memory consumption >> >> I am currently looking at a core of a httpd 2.4 process using the event MPM >> that consumed a lot of memory (core dump file size about 1.4 GB). >> While taking the core actually no request was processed by this process. >> I suspect a memory leak in a 3rd party closed source module. >> As I have no view in the 3rd party module I tried to find out the heap >> memory usage of httpd and the included modules. >> >> For this I used the new dump_pool_and_children I added to .gdbinit which >> delivers me the memory used by all pools below 'apr_global_pool' and the >> amount of memory in the allocator free lists associated to these pools. >> This delivered a usage of only a few MB. Of course I am aware that the >> process consumes additional memory for stack, static data and text >> segments, but this usage should be static. >> >> Is there any heap usage by httpd that I could have missed? >> >> - We do not use the unmanaged pools from APR. So I should have caught all >> pools. >> - I do no think that we use allocators that are not used by a pool. Hence I >> should have caught all allocators and >> its free lists. >> - As no request was processed when I captured the core (all worker threads >> were waiting for work) I doubt that missed >> any or at least large memory consuming buckets. > > Is APR configured to return memory to the OS? Otherwise you might just see > all the allocated memory in the free list of the pool allocator(s). > MaxMemFree is set to 2 MB in httpd and the allocator free list's that I have checked do not contain much memory (around 400 KB in total for all of them). Regards Rüdiger
Re: httpd memory consumption
On 10/06/2017 10:12 AM, Joe Orton wrote: > On Fri, Oct 06, 2017 at 09:46:39AM +0200, Ruediger Pluem wrote: >> For this I used the new dump_pool_and_children I added to .gdbinit which >> delivers me the memory used by all >> pools below 'apr_global_pool' and the amount of memory in the allocator free >> lists associated to these pools. >> This delivered a usage of only a few MB. Of course I am aware that the >> process consumes additional memory for stack, >> static data and text segments, but this usage should be static. > > Very nice! > >> >> Is there any heap usage by httpd that I could have missed? > > OpenSSL? Good pointer. Possibly not the case here as the same openssl version is used with other similar setups, but I have no hard prove for that assumption. Any pointers how to gather the memory usage of openssl from the core dump? Regards Rüdiger
RE: httpd memory consumption
> -Original Message- > From: Ruediger Pluem [mailto:rpl...@apache.org] > Sent: vrijdag 6 oktober 2017 09:47 > To: Apache HTTP Server Development List > Subject: httpd memory consumption > > I am currently looking at a core of a httpd 2.4 process using the event MPM > that consumed a lot of memory (core dump file size about 1.4 GB). > While taking the core actually no request was processed by this process. > I suspect a memory leak in a 3rd party closed source module. > As I have no view in the 3rd party module I tried to find out the heap > memory usage of httpd and the included modules. > > For this I used the new dump_pool_and_children I added to .gdbinit which > delivers me the memory used by all pools below 'apr_global_pool' and the > amount of memory in the allocator free lists associated to these pools. > This delivered a usage of only a few MB. Of course I am aware that the > process consumes additional memory for stack, static data and text > segments, but this usage should be static. > > Is there any heap usage by httpd that I could have missed? > > - We do not use the unmanaged pools from APR. So I should have caught all > pools. > - I do no think that we use allocators that are not used by a pool. Hence I > should have caught all allocators and > its free lists. > - As no request was processed when I captured the core (all worker threads > were waiting for work) I doubt that missed > any or at least large memory consuming buckets. Is APR configured to return memory to the OS? Otherwise you might just see all the allocated memory in the free list of the pool allocator(s). Bert
Re: httpd memory consumption
On Fri, Oct 06, 2017 at 09:46:39AM +0200, Ruediger Pluem wrote: > For this I used the new dump_pool_and_children I added to .gdbinit which > delivers me the memory used by all > pools below 'apr_global_pool' and the amount of memory in the allocator free > lists associated to these pools. > This delivered a usage of only a few MB. Of course I am aware that the > process consumes additional memory for stack, > static data and text segments, but this usage should be static. Very nice! > > Is there any heap usage by httpd that I could have missed? OpenSSL?
Re: httpd memory consumption
On 10/06/2017 09:56 AM, Stefan Eissing wrote: > Were many threads spawned? Asking for a friend... 17 by httpd and 12 by the 3rd party module. Additional information: no mod_http2 loaded :-) Regards Rüdiger > >> Am 06.10.2017 um 09:46 schrieb Ruediger Pluem : >> >> I am currently looking at a core of a httpd 2.4 process using the event MPM >> that consumed a lot of memory >> (core dump file size about 1.4 GB). >> While taking the core actually no request was processed by this process. >> I suspect a memory leak in a 3rd party closed source module. >> As I have no view in the 3rd party module I tried to find out the heap >> memory usage of httpd and the included modules. >> >> For this I used the new dump_pool_and_children I added to .gdbinit which >> delivers me the memory used by all >> pools below 'apr_global_pool' and the amount of memory in the allocator free >> lists associated to these pools. >> This delivered a usage of only a few MB. Of course I am aware that the >> process consumes additional memory for stack, >> static data and text segments, but this usage should be static. >> >> Is there any heap usage by httpd that I could have missed? >> >> - We do not use the unmanaged pools from APR. So I should have caught all >> pools. >> - I do no think that we use allocators that are not used by a pool. Hence I >> should have caught all allocators and >> its free lists. >> - As no request was processed when I captured the core (all worker threads >> were waiting for work) I doubt that missed >> any or at least large memory consuming buckets. >> >> So possible other consumers of memory apart from the 3rd party module that >> come to my mind: >> >> - Could any of the glibc (httpd was running on a Linux system) functions >> wrapped by APR consume a lot of memory? >> - As far as I researched glibc by default should return memory to the OS if >> more then 128 KB are free at the top of >> the heap. >> >> Regards >> >> Rüdiger > >
Re: httpd memory consumption
Were many threads spawned? Asking for a friend... > Am 06.10.2017 um 09:46 schrieb Ruediger Pluem : > > I am currently looking at a core of a httpd 2.4 process using the event MPM > that consumed a lot of memory > (core dump file size about 1.4 GB). > While taking the core actually no request was processed by this process. > I suspect a memory leak in a 3rd party closed source module. > As I have no view in the 3rd party module I tried to find out the heap memory > usage of httpd and the included modules. > > For this I used the new dump_pool_and_children I added to .gdbinit which > delivers me the memory used by all > pools below 'apr_global_pool' and the amount of memory in the allocator free > lists associated to these pools. > This delivered a usage of only a few MB. Of course I am aware that the > process consumes additional memory for stack, > static data and text segments, but this usage should be static. > > Is there any heap usage by httpd that I could have missed? > > - We do not use the unmanaged pools from APR. So I should have caught all > pools. > - I do no think that we use allocators that are not used by a pool. Hence I > should have caught all allocators and > its free lists. > - As no request was processed when I captured the core (all worker threads > were waiting for work) I doubt that missed > any or at least large memory consuming buckets. > > So possible other consumers of memory apart from the 3rd party module that > come to my mind: > > - Could any of the glibc (httpd was running on a Linux system) functions > wrapped by APR consume a lot of memory? > - As far as I researched glibc by default should return memory to the OS if > more then 128 KB are free at the top of > the heap. > > Regards > > Rüdiger
httpd memory consumption
I am currently looking at a core of a httpd 2.4 process using the event MPM that consumed a lot of memory (core dump file size about 1.4 GB). While taking the core actually no request was processed by this process. I suspect a memory leak in a 3rd party closed source module. As I have no view in the 3rd party module I tried to find out the heap memory usage of httpd and the included modules. For this I used the new dump_pool_and_children I added to .gdbinit which delivers me the memory used by all pools below 'apr_global_pool' and the amount of memory in the allocator free lists associated to these pools. This delivered a usage of only a few MB. Of course I am aware that the process consumes additional memory for stack, static data and text segments, but this usage should be static. Is there any heap usage by httpd that I could have missed? - We do not use the unmanaged pools from APR. So I should have caught all pools. - I do no think that we use allocators that are not used by a pool. Hence I should have caught all allocators and its free lists. - As no request was processed when I captured the core (all worker threads were waiting for work) I doubt that missed any or at least large memory consuming buckets. So possible other consumers of memory apart from the 3rd party module that come to my mind: - Could any of the glibc (httpd was running on a Linux system) functions wrapped by APR consume a lot of memory? - As far as I researched glibc by default should return memory to the OS if more then 128 KB are free at the top of the heap. Regards Rüdiger
Re: .gdbinit changes backports. CTR or RTC?
On 10/05/2017 05:07 PM, Yann Ylavic wrote: > On Thu, Oct 5, 2017 at 2:28 PM, Eric Covener wrote: >> On Thu, Oct 5, 2017 at 8:08 AM, Plüm, Rüdiger, Vodafone Group >> wrote: >>> Is backporting .gdbinit changes to a stable branch CTR or RTC? >> >> I would think CTR for a typical change there is reasonable. > > +1 r1811294, r1811295, r1811296 > > Nit: about commit, s/childs/children/ ? > r1811285 Regards Rüdiger