[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[
http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12322710
]
Graham Dumpleton commented on MODPYTHON-77:
---
I worked out what was going on a few days ago, but first time with Internet
access since. :-(
I agree that what is being proposed is reasonable provided that correct changes
which will work for any version of 2.3 or later versions of Python.
Some further clarification on the issue.
1. The new API does not really break the concept of multiple interpreters. What
it means is that if someone is lazy enough to use the new API they limit their
extension to only being usable with the first created interpreter. Someone can
still use the existing APIs and write the extension to work in a multi
interpreter context.
2. Even with the patches, an extension will still not work properly. In order
to work it is important that PythonImport and
PythonInterpreter be used as appropriate and that main_interpreter be
explicitly specified. This means that one can't have multiple interpreters for
different virtual hosts or applications which is a big shortcoming.
In short, extensions writers should be discouraged from using the new API if
want to work with mod_python.
Out ot time, gotta go.
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c,
mod_python.c.diff, mod_python.h.diff, src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
Re: Getting ready for 3.2 beta 2
I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Well, if I've understood Jim's mail, apart from the new MODPYTHON-77, we're all set. Regards, Nicolas2005/9/6, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I've been away this weekend - just got back, but I'm too busy to try toread all the multiple-interpreter related comments. I guess my question is- can someone provide a quick summary of how far we are from 3.2.1b testtarbal?Thanks!GrishaOn Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Gregory (Grisha) Trubetskoy wrote: I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? I've also been away for the weekend. The patch for MODPYTHON-77 from last Thursday was causing apache to segfault and I have not had a chance to try the lastest changes from Boyan. As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. Jim Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
On Tue, 6 Sep 2005, Jim Gallacher wrote: As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. +1 Grisha
