Where are we at with confirming these memory leaks and committing some
changes to fix them?
Only ask as I have a big patch related to MODPYTHON-63 that am wanting
to commit back into the repository, but don't want to be doing it if
it is going
to make your search harder by introducing some new memory leaks. :-)
BTW, what platform does you leak tester require to run? On Mac OS X I am
always seeing slow memory leaks over time even with most basic handler.
This often makes it hard to know if changes I make are introducing new
memory leaks or not. :-(
Graham
On 12/08/2006, at 7:55 AM, Jim Gallacher wrote:
Alexis Marrero wrote:
Where is the source code for _apache.make_table() ?
It's a rather convoluted path, but src/tableobject.c is where you
end up.
I've added a test to my leak test suite for apache.table().
(apache.make_table() was deprecated at some point). It does indeed
leak
like crazy.
Jim
Alexis Marrero wrote:
Jim,
I found the culprit!!!
There are two unrelated memory leaks.
The first one is in req_readline().
This code:
/* is there anything left in the rbuff from previous reads? */
if (self->rbuff_pos < self->rbuff_len) {
/* if yes, process that first */
while (self->rbuff_pos < self->rbuff_len) {
buffer[copied++] = self->rbuff[self->rbuff_pos];
if ((self->rbuff[self->rbuff_pos++] == '\n') ||
(copied == len)) {
/* our work is done */
/* resize if necessary */
if (copied < len)
if(_PyString_Resize(&result, copied))
return NULL;
return result;
}
}
}
Should look like this:
/* is there anything left in the rbuff from previous reads? */
if (self->rbuff_pos < self->rbuff_len) {
/* if yes, process that first */
while (self->rbuff_pos < self->rbuff_len) {
buffer[copied++] = self->rbuff[self->rbuff_pos];
if ((self->rbuff[self->rbuff_pos++] == '\n') ||
(copied == len)) {
/* our work is done */
/* resize if necessary */
if (copied < len)
if(_PyString_Resize(&result, copied))
return NULL;
if (self->rbuff_pos >= self->rbuff_len && self->rbuff
!= NULL)
{
free(self->rbuff);
self->rbuff = NULL;
}
return result;
}
}
}
That solves one. Like I mentioned in one of the emails to the
mailing
list, the buffer was not been freed in the last readline().
The second one, for which I don't have a fix yet is
apache.make_table() in mod_python/util.py line 152. If I comment
lines
152, 225, 227 you will see that memory doesn't grow. I will keep
investigating...
Until the next email.
/amn
Jim Gallacher wrote:
I ran my baseline test with 500k requests, and got the following:
(Note that all the figures will have an error of +/- 0.1)
baseline 500k requests 1.7%
So it would seem that there is not a specific problem in
readline, or my
test case is messed up. FYI here are my 2 handlers:
def baseline_handler(req):
req.content_type = 'text/plain'
req.write('ok baseline:')
return apache.OK
def readline_handler(req):
# the body of the request consists of
# '\n'.join([ 'a'*10 for i in xrange(0,10) ])
req.content_type = 'text/plain'
count = 0
while(1):
line = req.readline()
if not line:
break
count += 1
req.write('ok readline: %d lines read' % count)
return apache.OK
Jim
Jim Gallacher wrote:
I'll have some time to investigate this over the next couple of
days. I
ran my leaktest script for FieldStorage and readline, and
FieldStorage
certainly still leaks, but I'm not so sure about readline itself.
baseline 1k requests 1.2%
readline 500k requests 1.6%
fieldstorage 498k requests 10.1%
The memory consumption figures are for a machine with 512MB ram.
I'm running my baseline test with 500k requests right now to
see if the
1.6% figure for readline represents a real leak in that
function, or if
it is just mod_python itself.
My memory leak test suite is probably at the point that other
people
will find it useful. Once I've written a README explaining its
use I'll
commit it to the repository so everybody to play. If you anyone
wants to
give it a shot in the interim I can email it to you. Give me shout
offlist.
I haven't had a chance to look at the code you highlight below,
or at
least not closely. The whole req_readline function looks like
it will
require a good strong cup of coffee to fully comprehend. ;)
Jim
Alexis Marrero wrote:
Experimenting on this issue, I noticed that neither of the
following set
of "ifs" are ever met:
786 /* Free old rbuff as the old contents have been
copied
over and
787 we are about to allocate a new rbuff. Perhaps this
could
be reused
788 somehow? */
789 if (self->rbuff_pos >= self->rbuff_len && self->rbuff
!= NULL)
790 {
791 free(self->rbuff);
792 self->rbuff = NULL;
793 }
--------
846 /* Free rbuff if we're done with it */
847 if (self->rbuff_pos >= self->rbuff_len && self->rbuff
!= NULL)
848 {
849 free(self->rbuff);
850 self->rbuff = NULL;
851 }
I noticed that by putting some statements to write to the output
stream. They never execute.
/amn
On Aug 10, 2006, at 1:43 PM, Alexis Marrero wrote:
All,
We are trying to nail down a memory leak that happens only when
documents are POSTed to the server.
For testing we have a short script that does:
while True:
dictionary_of_parameters = {'field1': 'a'*100000}
post('url...', dictionary_of_parameters)
Then we run "top" on the server and watch the server memory grow
without bound. Why do we know that the problem is in
request.readline()? If I go to
mod_python.util.FieldStorage.read_to_boundary() and add the
following
statement:
def read_to_boundary(...):
return True
...
as the first executable line in the function the memory does not
grow.
I have read the req_readline a 1000 time and I can't figure
out where
the problem is.
My config:
Python 2.4.1
mod_python 3.2.10
Our request handler does nothing other than using
util.FieldStorage(req) and req.write('hello').
I have some suspicion that it has to do with:
....
19 * requestobject.c
20 *
21 * $Id: requestobject.c 420297 2006-07-09 13:53:06Z
nlehuen $
22 *
23 */
....
846 /* Free rbuff if we're done with it */
847 if (self->rbuff_pos >= self->rbuff_len && self-
>rbuff !=
NULL)
848 {
849 free(self->rbuff);
850 self->rbuff = NULL;
851 }
852
Though, I can't confirm.
/amn
