On Sun, 10 Sep 2000, Matt Sergeant wrote:
> On Sun, 10 Sep 2000, Stas Bekman wrote:
>
> > On Sun, 10 Sep 2000, Matt Sergeant wrote:
> >
> > > For 2 days solid now I've been trying to track down a very bizarre memory
> > > leak in AxKit.
> > >
> > > I've checked everything I can think of - all circular references are now
> > > gone, all closures clean up carefully after themselves, and I've reduced
> > > the usage of some external modules. But still the processes grow.
> > >
> > > Now to the wierd bit. I could track this down if it wasn't for this:
> > >
> > > The memory leak starts after the Nth hit, usually around 35. This is
> > > running under httpd -X.
> > >
> > > So it goes along very happily for 35 hits - memory usage is rock solid
> > > stable. Then after the 35th hit the memory usage starts to grow about 4k
> > > per hit. Obviously thats an impossible rate of growth to sustain for any
> > > amount of time, and soon the server is swamped.
> > >
> > > Can anyone give me _any_ help in figuring out where this might be coming
> > > from? I've tried several things, including adding the ability to display
> > > the memory usage in the debug statements (by getting the VmSize out of
> > > /proc/self/status). This is driving me absolutely mad, as you might
> > > tell. The only thing I can come close to in locating it is some remote
> > > possibility that it is in one particular section of the code, but that
> > > doesn't actually seem to be the case - sometimes the memory increase goes
> > > outside that section of code.
> > >
> > > The modules I'm using are:
> > >
> > > Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5,
> > > Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8,
> > > Unicode::String, MIME::Base64, Storable (loaded but not used),
> > > XML::Sablotron (loaded but not used).
> > >
> > > mod_perl is 1.24 and Perl is 5.00503 (Mandrake release).
> >
> > First, Apache::VMonitor and GTop will make your debugging work much
> > easier.
>
> Not really. :-(
I meant that GTop lets you an easier access to memory usage. Of course you
don't need it if you read directly the /proc/mem
> > Second, The only reason that I can think of that things start happening
> > after Nth hit is that one of the modules that you use does an internal
> > accounting that goes astrey after Nth hit. Using Apache::Peek will help
> > you discover the leakage by dumping and comparing the complete package
> > variables tables. You will find some examples of using Apache::Peek in the
> > guide. This will definitely allow you to spot the offending module. Of
> > course the complete list of loaded modules can be found in /perl-status :)
>
> I don't see this example. It doesn't seem like ::Peek gives me any useful
> information at all unless I recompile Perl with DEBUGGING_MSTATS, and that
> seems like a nightmare...
I was thinking about going thru the symbol table and dumping all the
variables. And run diff between the dumps of the two requests. Be careful
though that Devel::Peek doesn't show a complete dump for the whole
structure, and I couldn't find the interface for changing the deepness of
datastructure traversal. I think Doug has patched Apache::Peek to make
this configurable.
Do you think it still won't work?
> > Apache::Leak is another thing to try, but it's not really good unless you
> > know the exact offensive code. Otherwise it reports leakages for things
> > which are Perl optimizations in fact.
>
> I've tried ::Leak before and its not very helpful. Doesn't even give you a
> clue where the variables that are leaking are located.
Yup, that's what I've said :(
> > Yet another thing I'd think about is monitoring the code with GTop, and
> > running it under strace/gdb. So it goes like this:
> >
> > You run a daemon that watches the processes and prints its memory size via
> > Gtop. You complete 35 hits and now run the server under Apache::DB
> > debugger. So you do 's' by 's' (step by step) and watch the printouts of
> > the GTop daemon. Once you see the growth -- you know the offending
> > module/line - and voila... of course this should be automized...
>
> While I may now have started to hallucinate due to working on this
> non-stop for about 40 hours, this hasn't helped. The leak still moves
> around.
>
> I think maybe I have to go back to the drawing board on this one :-(
BTW, Frank's comment seems cool, I didn't think about it. What happens if
you create some small variable and start making it bigger (string?) and
see whether you still get this event on hit #36. May be you at least could
move it from #36 to #3. It's a documented fact that the data structures
change from hit #1 and #2, but not #3 assumming that it's the same
request.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com
http://singlesheaven.com http://perlmonth.com perl.org apache.org
