Re: two mod_python leaks

[Graham Dumpleton]

For such issues, by all means just send the email direct to the  
mailing list.
I have copied this back onto the list and I'll do some research into  
what
you describe.

BTW, I don't use Windows, but there are others on the list who do and it
may help if you can explain what methodology you are using to see the
Win32 handle leaks. This will help others to replicate it and then  
track it
down.
Thanks.

Graham

On 19/10/2006, at 8:50 PM, Jeff Robbins wrote:

Graham,

I apologize in advance for not understanding the email etiquette  
for the list, so I'm emailing you directly.  (I subscribed to the  
python-dev list and email the faq magic subject but the reply said  
"no info".)  Anyway...

I know of two leaks in Python 3.2.10 / Apache 2.2.3 that I would  
dearly love to get fixed.  One of them I have identified the source  
and made a change to my copy and it works (and comports with the  
python embedding documentation) so I was hoping you could either  
get it in or help me understand how I can get it applied.

The first leak is if you use the _repr_ function of a dict or list  
(which is as simple as using str([]) in your mod_python handler!)   
The repr functions use a dict in the python threadstate to avoid  
some recursion problem.  This dict is allocated only on demand, so  
while I found str([]) (or a str of any dict or list) to cause its  
allocation, there may be other user code things you can do to get  
it allocated.  To free it, the python embedder need to call  
"PyThreadState_Clear" before calling "PyThreadState_Delete".  Yet  
mod_python.c release_interpreter() does not call  
"PyThreadState_Clear".  The fixed version of release_interpreter is  
this:

static void release_interpreter(void)
{
PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
PyEval_ReleaseThread(tstate);
#else
PyThreadState_Swap(NULL);
#endif
// JSR
PyThreadState_Clear(tstate);
PyThreadState_Delete(tstate);
}



The Python embedded doc is clear that this is the correct thing to do:
http://docs.python.org/api/threads.html
...
       void PyThreadState_Clear( PyThreadState *tstate)

 Reset all information in a thread state object. The interpreter  
lock must be held.

       void PyThreadState_Delete( PyThreadState *tstate)

 Destroy a thread state object. The interpreter lock need not be  
held. The thread state must have been reset with a previous call to  
PyThreadState_Clear().
...





The second leak, unfortunately, I have not been able to fix.  It is  
in the parent httpd process on Win32 when you do a "restart" of  
apache.  mod_python causes the leak of 4 Win32 Events (which are  
likely allocated via use of various apr_ functions like the mutex  
functions)   This is reproducible and easily visible in the windows  
task manager if you look at the "handles" column and use either the  
command line or the apache monitor to restart apache.  On my  
installation, mod_python contributes 4 of the 5 leaking handles.   
(Someone else leaks one also!!!)  This harms a long-term deployment  
which uses the apache maxrequests feature to restart the child  
every million requests.

Thank you in advance for you patience with this long-winded email!

Best regards,

Jeff Robbins


----- Original Message ----- From: "Graham Dumpleton"  
<[EMAIL PROTECTED]>
To: <python-dev@httpd.apache.org>
Sent: Wednesday, October 18, 2006 20:47
Subject: Status of mod_python 3.3.


On JIRA, the following issues are still marked as incomplete for  
mod_python
version 3.3. I have noted my own comments about where they are up  
to and
what I think still needs to be done.

MODPYTHON-93 Improve util.FieldStorage efficiency.

This was actually marked as resolved but reopened because it was  
discovered
that changes meant that Trac <=0.9.6 would no longer work. The  
changes were
also backed out of mod_python 3.2.X branch and not released in  
3.2.10.

At this point I believe we have agreed that code in 3.3 would be  
left as is and
people would need to use Trac >=0.10, which has now been release,  
with
mod_python 3.3 or later.

There was comments as to whether util.FieldStorage needs to have more
dictionary like access, but at this point I believe we should mark  
this issue
as resolved and if people want dictionary like access, they can  
open a
separate JIRA issue for that and we deal with it in a future release.

In summary, I believe we should mark this as resolved.

MODPYTHON-104 Allow Python code callouts with mod_include (SSI).

The code for all this has been done for some time. The only reason  
it hasn't
been marked as resolved as no documentation has been added into core
mod_python documentation. I have separately written a article on  
the new
feature which is available at:

http://www.dscpl.com.au/wiki/ModPython/Articles/ 
BasicsOfServerSideIncludes

I have no problem as this being used as basis for core documentation.
I had been holding off integrating it because of contention over  
whether we
could use wiki for documentation or not.

In summary, need to still keep this open until some documentation  
added
to core mod_python documentation.

MODPYTHON-127 Use namespace for mod_python PythonOption settings.

I have made code changes but not committed them back to  
repository. Jim
has committed some documentation changes related to it already  
though.
Some more documentation changes are probably required where  
options are
mentioned in relation to features they affect.

I had posed question about whether  
mod_python.session.database_directory
should also be added as a general fallback in cases where which  
type of
filesystem based session was not going to be known. Jim responded  
with +1,
but his explicit vs implicit comment made me unsure which proposal he
was agreeing with. Thus nothing done about that yet.

In summary, bit more work to do.

MODPYTHON-143 Implement and integrate a new module importer.

Code has been done, except for extra bit more logging of  
exceptions for when
modules hooks are called and a problem occurs. Also need to update  
the
documentation.

In summary, more work to do but mainly documentation.

MODPYTHON-186 Build process not using correct values from Python
config Makefile.

This is only known to be an issue on Mac OS X when the very latest
compiler tool chain software given out at Mac developer conference
is used. It may only affect new Intel Macs. Someone did suggest they
would come back with required changes but that hasn't happened.

Personally I would say we don't attempt to address this in 3.3 and  
defer
it till later.

MODPYTHON-190 Python 2.5 support.

From what I have seen, people are already using Python 2.5, thus  
is there
any urgency on this? All I can figure is that by not making  
changes you will
not be able to work with really large data, or will it all crash  
badly on a 64
bit platform with 64 bit support compiled in.

I don't know the answers to this and no one else (doesn't even  
have to be
one of the committers) has stepped up to do the work and work out  
what
the required changes are.

Personally I would say we don't attempt to address this in 3.3 and  
defer
it till later.

MODPYTHON-193 Add req.hlist.location to mirror req.hlist.directory.

I have done most of the code for this and now just sorting out some
problems with trailing slashes getting added when they shouldn't.  
Also
need to still update Session code and ensure None is returned for  
handler
directory when not used in a directory.

In summary, a bit more work to do.

So that is where we are at. Can I get some agreement that we will:

1. Mark MODPYTHON-93 as resolved.
2. Defer MODPYTHON-186 and MODPYTHON-190 till later.

Finally, is there anything else in outstanding issues (not listed  
here)
that people believe need to be address for mod_python 3.3?

Graham