[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--
Fix Version: 3.3
Assign To: Graham Dumpleton
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Assignee: Graham Dumpleton
Fix For: 3.3
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz,
grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff,
src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Graham Dumpleton (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ] Graham Dumpleton updated MODPYTHON-77: -- Attachment: grahamd_20051105.tar.gz Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b. All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on restart and on Linux where Apache would crash after the request had been handled. Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same. The changes work fine on: Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed) Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5 Test example was gilstate.tar.gz attached to MODPYTHON-77. Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well. Failures? What failures? Jim
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--
Attachment: grahamd_20051105.tar.gz
Here is my first go at an alternate patch for this problem. Patch was made
against SVN head, believed to be 3.2.4b.
All the change in effect does is save the first interpreter as
main_interpreter, but most importantly it does this in Apache child process and
not in the parent process before the fork occurs as was the case before. This
avoids problems on Mac OS X where Apache would crash on restart and on Linux
where Apache would crash after the request had been handled.
Note that this change doesn't use any of the PEP GIL specific calls nor does it
do anything specific to make anything work on Python 2.3.5. Except for moving
one thread state swap call from the parent process context to the child process
context, all thread management code is the same.
The changes work fine on:
Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
Test example was gilstate.tar.gz attached to MODPYTHON-77.
Also passed on mod_python/test suite on Mac OS X. There were failures of test
suite on Linux, but those failures occurred before patches were applied as well.
The changes need to be tested on Win32 Apache as well as system where no thread
support compiled into Python.
For those of you following this issue, if you can test this change, indicate if
it works or fails and if it fails indicate specifically how it is failing. From
any failures can then start to understand the other changes Boyan has made and
what is required and what isn't.
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz,
grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff,
src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: mod_python.h.diff
mod_python.c.diff
mod_python.h.diff and mod_python.c.diff attached:
Changes:
1. Added some comments regarding the thread state changes
2. #ifdef PythonVer=235 because og the use of new Python APIs.
3. sequence of PyThreadState_Delete(tstate) and PyEval_ReleaseLock() changed -
the lock should be held while delete
So now the thread state handling should be completly compatible to Python =
2.3.5. and some issues should be fixed
e.g. creating thread from python handler will work without limitations for the
main interpreter and if you take
http://sourceforge.net/tracker/index.php?func=detailaid=1163563group_id=5470atid=105470
also for all other interpreters.
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c,
mod_python.c.diff, mod_python.h.diff, src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: gil_test.c
I hope that, the following can help clarifying where the problem comes from.
The root of the evil was the introduction of the new Python APIs
PyGILState_Ensure and PyGILState_Release. They are designed to make the process
of embedding Python and the extension development much easier, but they also
break the idea of having multiple interpreters. If you look at the
implementation of the PyGILState_Ensure you will see that the Python code
executed in a PyGILState_Ensure/PyGILState_Release scope is using the
autoInterpreterState, which is the one initialized by Py_Initialize. So you can
easily mix multiple interpreters in the same call stack... Another fact is that
more and more essential extensions are using those new Python APIs. That's why
I thought, that the most acceptable way of 'un-breaking' the concept of having
multiple interpreters and using complex Python extensions is:
- To specify, that the use of Python extensions can work only for
interpreters[MAIN_INTERPRETER]
- The interpreters[MAIN_INTERPRETER] must be the same as the
pystate.c-autoInterpreterState, so the one created by Py_Initiualize.
I've written a very simplified pace of code doing both embedding and extending
Python in order to exemplify what I'm talking about (see the attached
gil_test.c).
Case 1: Python - it works:
Python 2.3.5 (#62, Sep 2 2005, 19:32:54) [MSC v.1200 32 bit (Intel)] on win32
import gil_test
attribute_found = gil_test.funct1()
sys.testAttr = testAttr
print 'attribute_found =', attribute_found
attribute_found = 1
Case 2: mod_python (not patched) - does not work:
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
Case 3: mod_python (patched) - works
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: src.zip
Jim,
could you please try, if the cahnges in the attaced src.zip are working for
you. If so this will be a much better solution.
TIA
Boyan
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c,
src.zip
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: mod_python.c
changes ware made against 3.1.4
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: diff.txt
Diff aginst the latest version of mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: diff2.txt
Now I've attached the a diff file created with diff -u. Sorry for the
inconvenience.
Jim, please make sure that you have took the complete patch. May be you have
not removed the following lines from the python_init function:
-/* Release the thread state because we will never use
- * the main interpreter, only sub interpreters created later. */
-PyThreadState_Swap(NULL);
-
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-
Attachment: diff3.txt
It seems that diff2.txt was not transfered correctly to the server, so
diff3.txt attached.
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global
Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more
with Python = 2.3, because they are supposed to work only with the first
interpreter state initialized for the current process (a problem we
experienced). The first interpreter state is not used by mod_python after the
python_init is called.
One solution, which works fine for me, is to save the first interpreter state
into the interpreters dictionary in the function python_init
(MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock
Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
