Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
OK, I found the problem. The LockFile line is commented out in test.py which causes Apache to try to create the lock in the default location in /var/run. http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/test/test.py?rev=125771r1=106619r2=125771 So the question is - can we just put it back in, or does it break something on Win32? Cheers Grisha On Tue, 13 Sep 2005, Jim Gallacher wrote: Nicolas Lehuen wrote: Jim, do you manage to build and test the 3.1.4 version on your setup ? This looks like a permission problem, not something related to our current problem. I haven't tried 3.1.4. And I could also try the tests as root, which would eliminate any permission problems. I have a busy day ahead of me so this will have to wait until tonight. Jim Regards, Nicolas 2005/9/13, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: -1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.cMon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); -PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD -PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but nothing we couldn't handle :-) I did a little more research. Basically, this started with Graham's patch that addressed a problem with modules being reimported (or something). From Graham's message: The basic problem revolves around the Python dictionary used to hold the set of interpreters. The code in mod_python.c is trying to use the Python GIL to provide exclusive access to that dictionary and any subsequent creation of an interpreter. The only catch is that in creating a new interpreter, the Python core is, in someway I don't understand, swapping thread states at some point which is allowing other threads to acquire the GIL. So what Graham's patch does is create an APR lock (interpreters_lock) and wrap all the access to the dictionary with calls to apr_mutex_lock/unlock. I think the _real_ way to address this issue is to first find what is the problem with using the Python GIL to serialize access to the interpreters dictionary. Is this a Python bug, or are we not understanding GIL and using it improperly? BUT, given that the above question may be complicated to answer, and that Graham's patch resolves the issue, another thought: If the APR lock works, what is the point of using the GIL in addition? Should we just use the APR-based lock alone? I.e., where we had (after Graham's patch): #ifdef WITH_THREAD apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif we would use: #ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); #endif _without_ a call to PyEval_AcquireLock() at all. It should compile OK, and on platforms where APR has no thread support, like you said, it's not an issue since no separate interpreters run in one process at the same time. Grisha On Mon, 12 Sep 2005, Nicolas Lehuen wrote:
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
I'd keep the patch nonetheless - I think this is how it should be. Grisha On Mon, 12 Sep 2005, Jim Gallacher wrote: -1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.cMon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); -PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD -PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but nothing we couldn't handle :-) I did a little more research. Basically, this started with Graham's patch that addressed a problem with modules being reimported (or something). From Graham's message: The basic problem revolves around the Python dictionary used to hold the set of interpreters. The code in mod_python.c is trying to use the Python GIL to provide exclusive access to that dictionary and any subsequent creation of an interpreter. The only catch is that in creating a new interpreter, the Python core is, in someway I don't understand, swapping thread states at some point which is allowing other threads to acquire the GIL. So what Graham's patch does is create an APR lock (interpreters_lock) and wrap all the access to the dictionary with calls to apr_mutex_lock/unlock. I think the _real_ way to address this issue is to first find what is the problem with using the Python GIL to serialize access to the interpreters dictionary. Is this a Python bug, or are we not understanding GIL and using it improperly? BUT, given that the above question may be complicated to answer, and that Graham's patch resolves the issue, another thought: If the APR lock works, what is the point of using the GIL in addition? Should we just use the APR-based lock alone? I.e., where we had (after Graham's patch): #ifdef WITH_THREAD apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif we would use: #ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); #endif _without_ a call to PyEval_AcquireLock() at all. It should compile OK, and on platforms where APR has no thread support, like you said, it's not an issue since no separate interpreters run in one process at the same time. Grisha On Mon, 12 Sep 2005, Nicolas Lehuen wrote: Duh, this is becoming difficult :) I was thinking that if APR_HAS_THREADS was 0, then Apache was forcibly ran in prefork mode, so there was no need for thread safety at all, given the fact that mod_python would only run one interpreter thread. So if WITH_THREAD was not defined, ORAPR_HAS_THREADS was 0, then we would not need any thread safety code. Hence the definition of MOD_PYTHON_WITH_THREAD_SUPPORT. You're right in writing that a user could launch a new thread in Python, provided that WITH_THREAD is defined, even if APR_HAS_THREADS==0. However, having a look at the parts of mod_python.c where the thread safety was put in, I think we can safely say that those parts are only called by mod_python (through python_handler, python_cleanup etc who call get_interpreter). Those parts are therefore always called in the same thread (if APR_HAS_THREADS==0, that is) and there is no need for thread synchronization to be done (no shared data between the main thread and the other user threads, no need to release the GIL etc.). BUT, I could be very, very wrong here, and your idea of reverting to a conservative shield python threading calls with WITH_THREAD and apr threading code with APR_HAS_THREADS is way more attractive to my tired mind right now. So if you want I can revert all this MOD_PYTHON_WITH_THREAD_SUPPORT hack. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Done. All tests pass on my FreeBSD box. Nicolas - can you test Win32, I'm not 100% sure if the change to test.py I made will work. Grisha On Tue, 13 Sep 2005, Nicolas Lehuen wrote: Yes, now I remember I had to comment this line out because it broke something on Win32. Go ahead, uncomment it and I'll add a test to remove it when running the test on Win32. Regards, Nicolas 2005/9/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: OK, I found the problem. The LockFile line is commented out in test.py which causes Apache to try to create the lock in the default location in /var/run. http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/test/test.py?rev=125771r1=106619r2=125771 So the question is - can we just put it back in, or does it break something on Win32? Cheers Grisha On Tue, 13 Sep 2005, Jim Gallacher wrote: Nicolas Lehuen wrote: Jim, do you manage to build and test the 3.1.4 version on your setup ? This looks like a permission problem, not something related to our current problem. I haven't tried 3.1.4. And I could also try the tests as root, which would eliminate any permission problems. I have a busy day ahead of me so this will have to wait until tonight. Jim Regards, Nicolas 2005/9/13, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: -1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.c Mon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); - PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD - PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but nothing we couldn't handle :-) I did a little more research. Basically, this started with Graham's patch that addressed a problem with modules being reimported (or something). From Graham's message: The basic problem revolves around the Python dictionary used to hold the set of interpreters. The code in mod_python.c is trying to use the Python GIL to provide exclusive access to that dictionary and any subsequent creation of an interpreter. The only catch is that in creating a new interpreter, the Python core is, in someway I don't understand, swapping thread states at some point which is allowing other threads to acquire the GIL. So what Graham's patch does is create an APR lock (interpreters_lock) and wrap all the access to the dictionary with calls to apr_mutex_lock/unlock. I think the _real_ way to address this issue is to first find what is the problem with using the Python GIL to serialize access to the interpreters dictionary. Is this a Python bug, or are we not understanding GIL and using it improperly? BUT, given that the above question may be complicated to answer, and that Graham's patch resolves the issue, another thought: If the APR lock works, what is the point of using the GIL in addition? Should we just use the APR-based lock alone? I.e., where we had (after Graham's patch): #ifdef WITH_THREAD apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif we would use: #ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); #endif _without_ a call to PyEval_AcquireLock() at all. It should compile OK, and on platforms where APR has no thread support, like you said, it's not an issue since no separate interpreters run in one process at the same time. Grisha On Mon, 12 Sep 2005, Nicolas Lehuen wrote: Duh, this is becoming difficult :) I was thinking that if APR_HAS_THREADS was 0, then Apache was forcibly ran in prefork mode, so there was no need for thread safety at all, given the fact that mod_python would only run one interpreter thread. So if WITH_THREAD was not defined, ORAPR_HAS_THREADS was 0, then we would not need any thread safety
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Gregory (Grisha) Trubetskoy wrote: Done. All tests pass on my FreeBSD box. Nicolas - can you test Win32, I'm not 100% sure if the change to test.py I made will work. Good news. If the changes can be checked in and Nicolas can give a +1 on the Windows test then I'll be able to generate the next, and hopefully last, beta tonight. As an aside, it may be useful to have an option for test.py to preserve a copy of the apache logs. It would make troubleshooting the unit test failures much easier. Currently test.py deletes the logs after each unit test. Regards, Jim Grisha On Tue, 13 Sep 2005, Nicolas Lehuen wrote: Yes, now I remember I had to comment this line out because it broke something on Win32. Go ahead, uncomment it and I'll add a test to remove it when running the test on Win32. Regards, Nicolas 2005/9/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: OK, I found the problem. The LockFile line is commented out in test.py which causes Apache to try to create the lock in the default location in /var/run. http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/test/test.py?rev=125771r1=106619r2=125771 So the question is - can we just put it back in, or does it break something on Win32? Cheers Grisha On Tue, 13 Sep 2005, Jim Gallacher wrote: Nicolas Lehuen wrote: Jim, do you manage to build and test the 3.1.4 version on your setup ? This looks like a permission problem, not something related to our current problem. I haven't tried 3.1.4. And I could also try the tests as root, which would eliminate any permission problems. I have a busy day ahead of me so this will have to wait until tonight. Jim Regards, Nicolas 2005/9/13, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: -1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.c Mon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); - PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD - PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but nothing we couldn't handle :-) I did a little more research. Basically, this started with Graham's patch that addressed a problem with modules being reimported (or something). From Graham's message: The basic problem revolves around the Python dictionary used to hold the set of interpreters. The code in mod_python.c is trying to use the Python GIL to provide exclusive access to that dictionary and any subsequent creation of an interpreter. The only catch is that in creating a new interpreter, the Python core is, in someway I don't understand, swapping thread states at some point which is allowing other threads to acquire the GIL. So what Graham's patch does is create an APR lock (interpreters_lock) and wrap all the access to the dictionary with calls to apr_mutex_lock/unlock. I think the _real_ way to address this issue is to first find what is the problem with using the Python GIL to serialize access to the interpreters dictionary. Is this a Python bug, or are we not understanding GIL and using it improperly? BUT, given that the above question may be complicated to answer, and that Graham's patch resolves the issue, another thought: If the APR lock works, what is the point of using the GIL in addition? Should we just use the APR-based lock alone? I.e., where we had (after Graham's patch): #ifdef WITH_THREAD apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif we would use: #ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); #endif _without_ a call to PyEval_AcquireLock() at all. It should compile OK, and on platforms where APR has no thread support, like you said, it's
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Alright, cool. So I guess we're ready to roll the next tarfile now. Grisha On Tue, 13 Sep 2005, Nicolas Lehuen wrote: I'm sure it is required, even after fixing the S in APR_HAS_THREADS I tried with and without the PyEval_AcquireLock code and the latter works while the former doesn't. I don't know why though... I'll have to review the code once more to understand. Regards, Nicolas 2005/9/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Good job - I've tested it and it works on FreeBSD. Just one last thing I wanted to confirm - are you sure that PyEval_AcquireLock is required, or was this a sideffect of the missing 'S'? I just want to make sure we're not doing double locking where it's not needed. Grisha On Tue, 13 Sep 2005, Nicolas Lehuen wrote: OK I fixed the problem, please check again on FreeBSD. Here is what I've done (from the subversion comment) : Fixed : APR_HAS_THREADS ends with an 'S'. Reintroduced the calls to PyEval_AcquireLock and PyEval_ReleaseLock as they are required if threading is enabled. Removed two debugging log entries. Added the conditional commenting of the LockFile directive in the LockFile class. Regards, Nicolas 2005/9/13, Nicolas Lehuen [EMAIL PROTECTED]: Well, bad news... I have a -1 for now on Win32. I don't know why, but when I install and test the 20050911 version, everything is OK. When I install and test the latest version, the unit test behave very strangely (with or without the changes done by Graham). Basically, the Apache server takes forever to stop. Weird... I'm trying to find out what happens here. Regards, Nicolas 2005/9/13, Jim Gallacher [EMAIL PROTECTED]: Gregory (Grisha) Trubetskoy wrote: Done. All tests pass on my FreeBSD box. Nicolas - can you test Win32, I'm not 100% sure if the change to test.py I made will work. Good news. If the changes can be checked in and Nicolas can give a +1 on the Windows test then I'll be able to generate the next, and hopefully last, beta tonight. As an aside, it may be useful to have an option for test.py to preserve a copy of the apache logs. It would make troubleshooting the unit test failures much easier. Currently test.py deletes the logs after each unit test. Regards, Jim Grisha On Tue, 13 Sep 2005, Nicolas Lehuen wrote: Yes, now I remember I had to comment this line out because it broke something on Win32. Go ahead, uncomment it and I'll add a test to remove it when running the test on Win32. Regards, Nicolas 2005/9/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: OK, I found the problem. The LockFile line is commented out in test.py which causes Apache to try to create the lock in the default location in /var/run. http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/test/test.py?rev=125771r1=106619r2=125771 So the question is - can we just put it back in, or does it break something on Win32? Cheers Grisha On Tue, 13 Sep 2005, Jim Gallacher wrote: Nicolas Lehuen wrote: Jim, do you manage to build and test the 3.1.4 version on your setup ? This looks like a permission problem, not something related to our current problem. I haven't tried 3.1.4. And I could also try the tests as root, which would eliminate any permission problems. I have a busy day ahead of me so this will have to wait until tonight. Jim Regards, Nicolas 2005/9/13, Jim Gallacher [EMAIL PROTECTED] mailto: [EMAIL PROTECTED]: -1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.c Mon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); - PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD - PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
OK, so on a non-threaded Apache, we can suppose we will be using the prefork MPM, so we don't need any code to support threading in mod_python, then, right ? In this case instead of testing for WITH_THREAD in mod_python.c : #ifdef WITH_THREAD maybe we could test for WITH_THREAD and APR_HAS_THREADS : #if APR_HAS_THREADS defined(WITH_THREAD) Right ? This would remove all threading-related code from mod_python when only prefork is available or when Python isn't compiled to support threads (I which case I wonder how it works in a threaded Apache...). I have given up using QEMU for now, minotaur is sufficient to make sure mod_python builds on FreeBSD. Granted, it won't allow me to give any +1 since I cannot run the unit tests (or can I ?)... Regards, Nicolas 2005/9/11, Jim Gallacher [EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD.If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads.I'm also setting up FreeBSD under QEMU... so far so good, but installinganything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used thebinary packages for the software I wanted to install. :-(Regards,JimJim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto: [EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer, but that will be long and troublesome. If someone has more clue on this issue, feel free to tell us :). Regards, Nicolas 2005/9/10, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]:I'm stubling around in the dark here, but maybe this will create a spark of an idea. I took a diff of mod_python.c from 3.1.4 and 3.2.1b andisolated the lines which correspond to the compilation error.Compiler messages -mod_python.c:34: error: syntax error before '*' tokenmod_python.c:34: warning: type defaults to `int' in declaration of`interpreters_lock' mod_python.c:34: warning: data definition has no type or storage classmod_python.c: In function `get_interpreter':mod_python.c:131: warning: implicit declaration of function `apr_thread_mutex_lock'mod_python.c:161: warning: implicit declaration of function`apr_thread_mutex_unlock'mod_python.c: In function `python_init': mod_python.c:517: warning: implicit declaration of function`apr_thread_mutex_create'mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (firstuse in this function) Diff output---I've only copied the diff chunks which correspond to the complier errors mentioned above.--- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005+++ mod_python-3.2.1b/src/mod_python.cTue Sep6 17:11:03 2005@@ -31,6 +31,8 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL;+static apr_thread_mutex_t* interpreters_lock = 0;+ apr_pool_t *child_init_pool = NULL; ... snip ...@@ -124,11 +128,15 @@ name = MAIN_INTERPRETER; #ifdef WITH_THREAD+apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif... snip ...@@ -149,6 +158,7 @@ #ifdef WITH_THREAD PyEval_ReleaseLock();+apr_thread_mutex_unlock(interpreters_lock); #endif... snip ...@@ -490,13 +506,15 @@ } /* initialize global Python interpreter if necessary */-if (! Py_IsInitialized())+if (initialized == 0 || !Py_IsInitialized()) {-+initialized = 1;+ /* initialze the interpreter */ Py_Initialize(); #ifdef WITH_THREAD+
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Nicolas Lehuen wrote: OK, so on a non-threaded Apache, we can suppose we will be using the prefork MPM, so we don't need any code to support threading in mod_python, then, right ? Makes sense to me. In this case instead of testing for WITH_THREAD in mod_python.c : #ifdef WITH_THREAD maybe we could test for WITH_THREAD and APR_HAS_THREADS : #if APR_HAS_THREADS defined(WITH_THREAD) Right ? This would remove all threading-related code from mod_python when only prefork is available or when Python isn't compiled to support threads (I which case I wonder how it works in a threaded Apache...). Seems reasonable. It compiles and passes the unit tests on debian. I have given up using QEMU for now, minotaur is sufficient to make sure mod_python builds on FreeBSD. Granted, it won't allow me to give any +1 since I cannot run the unit tests (or can I ?)... I got FreeBSD running under QEMU, and the results are the same as on minotaur. With the #if APR_HAS_THREADS defined(WITH_THREAD) code change, mod_python compiles but the final link step generates a warning on both minotaur and qemu: /usr/local/share/apache2/build/libtool --silent --mode=link cc -o mod_python.la -rpath /usr/local/libexec/apache2 -module -avoid-version hlistobject.lo hlist.lo filterobject.lo connobject.lo serverobject.lo util.lo tableobject.lo requestobject.lo _apachemodule.lo mod_python.lo -Wl,--export-dynamic -pthread -lm /usr/local/lib/python2.4/config/libpython2.4.a -lutil -lm *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! Likewise, the unit tests on both QEMU and minotaur fail. On minotaur: Syntax error on line 44 of /home/jgallacher/tmp/mod_python/test/conf/test.conf: Cannot load /home/jgallacher/tmp/mod_python/src/mod_python.so into server: /home/jgallacher/tmp/mod_python/src/mod_python.so: Undefined symbol btowc On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ?
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer, but that will
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Well, it depends : #if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1 #else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0 #endif It's not only a matter of Python supporting threads, we must also have a thread-enabled APR. So that's the reason for the weird name I chose. But I don't mind changing it ! Regards, Nicolas2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD?GrishaOn Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] : FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2 . The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
I'm not sure I understand this, perhaps someone could write a message to the list explaining what we're doing here so there is a record. Sorry if I'm being slow-headed here. To me it seems that when you use thread-related calls from Python, you wrap those in Python defines (WITH_THREAD) and when you use thread-related calls from APR, you wrap those in APR defines (APR_HAS_THREAD), and that's all? In other words - what does MOD_PYTHON_WITH_THREAD_SUPPORT accomplish that the above does not. Also, given: #if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1 #else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0 #endif Does this mean that if Python is compiled with thread support and APR is not, MOD_PYTHON_WITH_THREAD_SUPPORT is 0 which means that the thread safety code isn't there, but you still _can_ create threads because Python will let you - isn't this asking for a segfault/deadlock/whatever? Grisha On Mon, 12 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork,
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Duh, this is becoming difficult :) I was thinking that if APR_HAS_THREADS was 0, then Apache was forcibly ran in prefork mode, so there was no need for thread safety at all, given the fact that mod_python would only run one interpreter thread. So if WITH_THREAD was not defined, ORAPR_HAS_THREADS was 0, then we would not need any thread safety code. Hence the definition of MOD_PYTHON_WITH_THREAD_SUPPORT. You're right in writing that a user could launch a new thread in Python, provided that WITH_THREAD is defined, even if APR_HAS_THREADS==0. However, having a look at the parts of mod_python.c where the thread safety was put in, I think we can safely say that those parts are only called by mod_python (through python_handler, python_cleanup etc who call get_interpreter). Those parts are therefore always called in the same thread (if APR_HAS_THREADS==0, that is) and there is no need for thread synchronization to be done (no shared data between the main thread and the other user threads, no need to release the GIL etc.). BUT, I could be very, very wrong here, and your idea of reverting to a conservative shield python threading calls with WITH_THREAD and apr threading code with APR_HAS_THREADS is way more attractive to my tired mind right now. So if you want I can revert all this MOD_PYTHON_WITH_THREAD_SUPPORT hack. Regards, Nicolas2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I'm not sure I understand this, perhaps someone could write a message tothe list explaining what we're doing here so there is a record. Sorry ifI'm being slow-headed here.To me it seems that when you use thread-related calls from Python, you wrap those in Python defines (WITH_THREAD) and when you use thread-relatedcalls from APR, you wrap those in APR defines (APR_HAS_THREAD), and that'sall?In other words - what does MOD_PYTHON_WITH_THREAD_SUPPORT accomplish that the above does not.Also, given:#if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1#else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0#endif Does this mean that if Python is compiled with thread support and APR isnot, MOD_PYTHON_WITH_THREAD_SUPPORT is 0 which means that the threadsafety code isn't there, but you still _can_ create threads because Python will let you - isn't this asking for a segfault/deadlock/whatever?GrishaOn Mon, 12 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
-1 for this patch. Actually, the patch itself is fine - it just doesn't fix the problem. The unit tests are still failings as per my previous messages. ie the following is getting logged in test/logs/error_log: [Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory: Couldn't create accept lock Jim Gregory (Grisha) Trubetskoy wrote: Here's a patch (this is against 3.1.2b). Untested. This replaces GIL with with the APR lock. --- src/mod_python.c.orig Mon Sep 12 16:42:28 2005 +++ src/mod_python.cMon Sep 12 17:32:26 2005 @@ -31,7 +31,9 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +#ifdef APR_HAS_THREAD static apr_thread_mutex_t* interpreters_lock = 0; +#endif apr_pool_t *child_init_pool = NULL; @@ -127,9 +129,8 @@ if (! name) name = MAIN_INTERPRETER; -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); -PyEval_AcquireLock(); #endif if (!interpreters) { @@ -156,8 +157,7 @@ idata = (interpreterdata *)PyCObject_AsVoidPtr(p); } -#ifdef WITH_THREAD -PyEval_ReleaseLock(); +#ifdef APR_HAS_THREAD apr_thread_mutex_unlock(interpreters_lock); #endif @@ -513,8 +513,10 @@ /* initialze the interpreter */ Py_Initialize(); -#ifdef WITH_THREAD +#ifdef APR_HAS_THREAD apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); +#endif +#ifdef WITH_THREAD /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: Yep, this is getting a little hairy, but nothing we couldn't handle :-) I did a little more research. Basically, this started with Graham's patch that addressed a problem with modules being reimported (or something). From Graham's message: The basic problem revolves around the Python dictionary used to hold the set of interpreters. The code in mod_python.c is trying to use the Python GIL to provide exclusive access to that dictionary and any subsequent creation of an interpreter. The only catch is that in creating a new interpreter, the Python core is, in someway I don't understand, swapping thread states at some point which is allowing other threads to acquire the GIL. So what Graham's patch does is create an APR lock (interpreters_lock) and wrap all the access to the dictionary with calls to apr_mutex_lock/unlock. I think the _real_ way to address this issue is to first find what is the problem with using the Python GIL to serialize access to the interpreters dictionary. Is this a Python bug, or are we not understanding GIL and using it improperly? BUT, given that the above question may be complicated to answer, and that Graham's patch resolves the issue, another thought: If the APR lock works, what is the point of using the GIL in addition? Should we just use the APR-based lock alone? I.e., where we had (after Graham's patch): #ifdef WITH_THREAD apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif we would use: #ifdef APR_HAS_THREAD apr_thread_mutex_lock(interpreters_lock); #endif _without_ a call to PyEval_AcquireLock() at all. It should compile OK, and on platforms where APR has no thread support, like you said, it's not an issue since no separate interpreters run in one process at the same time. Grisha On Mon, 12 Sep 2005, Nicolas Lehuen wrote: Duh, this is becoming difficult :) I was thinking that if APR_HAS_THREADS was 0, then Apache was forcibly ran in prefork mode, so there was no need for thread safety at all, given the fact that mod_python would only run one interpreter thread. So if WITH_THREAD was not defined, ORAPR_HAS_THREADS was 0, then we would not need any thread safety code. Hence the definition of MOD_PYTHON_WITH_THREAD_SUPPORT. You're right in writing that a user could launch a new thread in Python, provided that WITH_THREAD is defined, even if APR_HAS_THREADS==0. However, having a look at the parts of mod_python.c where the thread safety was put in, I think we can safely say that those parts are only called by mod_python (through python_handler, python_cleanup etc who call get_interpreter). Those parts are therefore always called in the same thread (if APR_HAS_THREADS==0, that is) and there is no need for thread synchronization to be done (no shared data between the main thread and the other user threads, no need to release the GIL etc.). BUT, I could be very, very wrong here, and your idea of reverting to a conservative shield python threading calls with WITH_THREAD and apr threading code with APR_HAS_THREADS is way more attractive to my tired mind right now. So if you want I can revert all this MOD_PYTHON_WITH_THREAD_SUPPORT hack. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I'm not sure I understand this, perhaps someone could write a message to the list explaining what
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer, but that will be long and troublesome. If someone has more clue on this issue, feel free to tell us :). Regards, Nicolas2005/9/10, Jim Gallacher [EMAIL PROTECTED]: I'm stubling around in the dark here, but maybe this will create a spark of an idea. I took a diff of mod_python.c from 3.1.4 and 3.2.1b and isolated the lines which correspond to the compilation error. Compiler messages - mod_python.c:34: error: syntax error before '*' token mod_python.c:34: warning: type defaults to `int' in declaration of `interpreters_lock' mod_python.c:34: warning: data definition has no type or storage class mod_python.c: In function `get_interpreter': mod_python.c:131: warning: implicit declaration of function `apr_thread_mutex_lock' mod_python.c:161: warning: implicit declaration of function `apr_thread_mutex_unlock' mod_python.c: In function `python_init': mod_python.c:517: warning: implicit declaration of function `apr_thread_mutex_create' mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (first use in this function) Diff output --- I've only copied the diff chunks which correspond to the complier errors mentioned above. --- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005 +++ mod_python-3.2.1b/src/mod_python.cTue Sep6 17:11:03 2005 @@ -31,6 +31,8 @@* (In a Python dictionary) */ static PyObject * interpreters = NULL; +static apr_thread_mutex_t* interpreters_lock = 0; + apr_pool_t *child_init_pool = NULL; ... snip ... @@ -124,11 +128,15 @@ name = MAIN_INTERPRETER; #ifdef WITH_THREAD +apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif ... snip ... @@ -149,6 +158,7 @@ #ifdef WITH_THREAD PyEval_ReleaseLock(); +apr_thread_mutex_unlock(interpreters_lock); #endif ... snip ... @@ -490,13 +506,15 @@ } /* initialize global Python interpreter if necessary */ -if (! Py_IsInitialized()) +if (initialized == 0 || !Py_IsInitialized()) { - +initialized = 1; + /* initialze the interpreter */ Py_Initialize(); #ifdef WITH_THREAD + apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif So it would seem that the code causing the compile problems is new for 3.2. I also notice that in apr_arch_thread_mutex.h the typedef for apr_thread_mutex_t is wrapped by #if APR_HAS_THREADS / #endif. Looking at the apache source in srclib/apr/locks/unix/thread_mutex.c, everything is also enclosed by #if APR_HAS_THREADS / #endif. eg, apr_thread_mutex_create, apr_thread_mutex_lock and apr_thread_mutex_unlock. Hopefully this will give someone a clue as to what may be going on here with FreeBSD. Regards, Jim
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer, but that will be long and troublesome. If someone has more clue on this issue, feel free to tell us :). Regards, Nicolas 2005/9/10, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: I'm stubling around in the dark here, but maybe this will create a spark of an idea. I took a diff of mod_python.c from 3.1.4 and 3.2.1b and isolated the lines which correspond to the compilation error. Compiler messages - mod_python.c:34: error: syntax error before '*' token mod_python.c:34: warning: type defaults to `int' in declaration of `interpreters_lock' mod_python.c:34: warning: data definition has no type or storage class mod_python.c: In function `get_interpreter': mod_python.c:131: warning: implicit declaration of function `apr_thread_mutex_lock' mod_python.c:161: warning: implicit declaration of function `apr_thread_mutex_unlock' mod_python.c: In function `python_init': mod_python.c:517: warning: implicit declaration of function `apr_thread_mutex_create' mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (first use in this function) Diff output --- I've only copied the diff chunks which correspond to the complier errors mentioned above. --- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005 +++ mod_python-3.2.1b/src/mod_python.c Tue Sep 6 17:11:03 2005 @@ -31,6 +31,8 @@ * (In a Python dictionary) */ static PyObject * interpreters = NULL; +static apr_thread_mutex_t* interpreters_lock = 0; + apr_pool_t *child_init_pool = NULL; ... snip ... @@ -124,11 +128,15 @@ name = MAIN_INTERPRETER; #ifdef WITH_THREAD +apr_thread_mutex_lock(interpreters_lock); PyEval_AcquireLock(); #endif ... snip ... @@ -149,6 +158,7 @@ #ifdef WITH_THREAD PyEval_ReleaseLock(); +apr_thread_mutex_unlock(interpreters_lock); #endif ... snip ... @@ -490,13 +506,15 @@ } /* initialize global Python interpreter if necessary */ -if (! Py_IsInitialized()) +if (initialized == 0 || !Py_IsInitialized()) { - +initialized = 1; + /* initialze the interpreter */ Py_Initialize(); #ifdef WITH_THREAD + apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); /* create and acquire the interpreter lock */ PyEval_InitThreads(); #endif So it would seem that the code causing the compile problems is new for 3.2. I also notice that in apr_arch_thread_mutex.h the typedef for apr_thread_mutex_t is wrapped by #if APR_HAS_THREADS / #endif. Looking at the apache source in srclib/apr/locks/unix/thread_mutex.c, everything is also enclosed by #if APR_HAS_THREADS /
Re: Getting ready for 3.2 beta 2
I've tried to build 3.1.4 from the tarball on minotaur and of course it works. Could it be possible that the recent changes in the configure script cause the problem ? Regards, Nicolas 2005/9/10, Jim Gallacher [EMAIL PROTECTED]: I thought I'd it a shot on minotaur as well. Poking around a bit reveals that the default apache is indeed 1.3. It looks like there might be a viable apache2 hiding in /usr/local/apache2-install/www.apache.org/current. eg ./configure --with-apxs=/usr/local/apache2-install/www.apache.org/current/apxs Unfortunately, I'm getting the same errors as Grisha reported starting with: mod_python.c:34: error: syntax error before '*' token Regards, Jim Nicolas Lehuen wrote: I tried to build it under minotaur as well, but ./configure only finds a 1.3.33 version of Apache, so I can't go further. I can't help much here since I'm not used to FreeBSD... Regards, Nicolas 2005/9/9, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just tried compiling it on minotaur and I get the same error. minotaur is FreeBSD 5.4, so it looks like we have a -1. I don't know how much time I'll have this weekend, so I might or might not look into the cause of this - but anyone else with access to a FreeBSD box, you're more than welcome to dig in... :-) Grisha On Fri, 9 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Don't know about versions, but I'd _really_ like to see a FreeBSD +1 at this point :-) Graham - don't you have FreeBSD access somewhere? If Graham can't help out maybe we could recruit a volunteer on the mod_python list? On the versioning discussion - I don't like 4.0, I think 3.3 should be the next version after 3.2.x. As far as even/odd stable/unstable - the Linux kernel folks have abandoned it because it didn't work for them. The fallacy is that you cannot know ahead of time what is stable and what is not. My preference is to just follow versions incrementally, and making it known which version is stable or not independently of the version number, which is what the HTTPD folks have been doing. I can't get worked up one way or another wrt to a version numbering scheme, as long as we release *something*. ;) Regards, Jim
Re: Getting ready for 3.2 beta 2
Jim Gallacher wrote .. Gregory (Grisha) Trubetskoy wrote: I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? I've also been away for the weekend. The patch for MODPYTHON-77 from last Thursday was causing apache to segfault and I have not had a chance to try the lastest changes from Boyan. As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If we feel that 3.3 could be a while in coming, I'm tending to think that this change to support use of extensions using the simplified GIL interface should be incorporated into 3.2. This would depend though on how many more beta snapshots we think we might go through. Graham
Re: Getting ready for 3.2 beta 2
2005/9/8, Jorey Bump [EMAIL PROTECTED]: Jim Gallacher wrote: Nicolas Lehuen wrote: Well, why not keep our plan of releasing 3.2 ASAP and save this problem for a later 3.2.x as a bug fix ? Making subsequent bug-fix releases should be fast and easy. We cannot afford to repeat the long hiatus between 3.1.3 and 3.2, with a long period of time without any official bug fix. I agree that 3.3 may come later, but we definitely should be able to release 3.2 bugfixes version as often as possible. This will save us and our users a lot of time, allowing us to stop writing yeah, we know this bug, it's already fixed in SVN but you'll have to wait an undefinite time for the fix to go public. +1 It's always tempting to make one last change, fix one more bug, but then the release never happens. I think everyone has the will to move mod_python forward, we just need a little more discipline. There are lots of things we can do in 3.3, but I for one am not motivated to work on these until 3.2 is out. Lets get this puppy out the door and then have a discussion on plans and priorities for 3.3 with a view to reducing the time between bug fixes and major releases. Would it help to adopt a naming convention where odd minor versions are for development, and even minor versions are stable/bug-fix-only? This would be a convenient time to adopt it. In some environments, this gives developers a place to add new features (3.3.x) while the first stable release (3.2.0) is getting bug squashed. As a user, it makes things a lot clearer that a certain version is still in development when you lust after a new feature it offers. Just a thought... Yeah, why not. In any case, we should maintain a separate 3.2 branch with only bug fixes while developing the 3.3 version on the trunk (and merge the bugfixes from the 3.2 version into the 3.3 trunk, of course). We haven't done this for the 3.1 and 3.2 version, so everybody will need to upgrade to 3.2 even if they want a single bugfix. This is not a really bad thing this time, but next time, if we start to fool around and break some compatibility (think new import system here :), we should make sure we don't force our users to upgrade just to get one bugfix. Regards, Nicolas
Re: Getting ready for 3.2 beta 2
On 09/09/2005, at 10:02 AM, Jim Gallacher wrote: As far as some future version breaking compatibility, I favour a bigger jump in the major number: 3.2 - 4.0. This is server software after all, and some people may prefer to maintain an older version for a longer period, foregoing new features in favour of the tried and true. Incrementing the major number makes it more obvious that an upgrade may cause some problems. But I guess that discussion is sometime *way* in the future. :) I also have been thinking that a jump to version 4.0 would be better for what is being speculated on for the next release. Graham
Re: Getting ready for 3.2 beta 2
I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Well, if I've understood Jim's mail, apart from the new MODPYTHON-77, we're all set. Regards, Nicolas2005/9/6, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I've been away this weekend - just got back, but I'm too busy to try toread all the multiple-interpreter related comments. I guess my question is- can someone provide a quick summary of how far we are from 3.2.1b testtarbal?Thanks!GrishaOn Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
Gregory (Grisha) Trubetskoy wrote: I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? I've also been away for the weekend. The patch for MODPYTHON-77 from last Thursday was causing apache to segfault and I have not had a chance to try the lastest changes from Boyan. As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. Jim Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
Re: Getting ready for 3.2 beta 2
On Tue, 6 Sep 2005, Jim Gallacher wrote: As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. +1 Grisha
Re: Getting ready for 3.2 beta 2
Or speaking in diff (not tested): --- setup.py.in.orig2005-09-01 11:42:09.082202944 -0400 +++ setup.py.in 2005-09-01 11:44:35.969872624 -0400 @@ -140,18 +140,24 @@ # this is a hack to prevent build_ext from trying to append initmod_python to the export symbols self.export_symbols = finallist(self.export_symbols) -ModPyModule = ModPyExtension(getmp_srcdir(), [getmp_includedir(), getapache_includedir()], [getapache_libdir()]) if winbuild: + +# build mod_python.so +ModPyModule = ModPyExtension(getmp_srcdir(), [getmp_includedir(), getapache_includedir()], [getapache_libdir()]) + scripts = [win32_postinstall.py] # put the mod_python.so file in the Python root ... # win32_postinstall.py will pick it up from there... # data_files = [(, [(os.path.join(getmp_srcdir(), 'Release', 'mod_python.so'))])] data_files = [] +ext_modules = [ModPyModule, PSPModule] + else: -# mpso = ../src/mod_python.so + scripts = [] data_files = [] +ext_modules = [PSPModule] import string from distutils import sysconfig @@ -174,7 +180,7 @@ package_dir={'mod_python': os.path.join(getmp_rootdir(), 'lib', 'python', 'mod_python')}, scripts=scripts, data_files=data_files, - ext_modules=[ModPyModule, PSPModule]) + ext_modules=ext_modules) # makes emacs go into python mode ### Local Variables: On Thu, 1 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Wed, 31 Aug 2005, Jim Gallacher wrote: 3. Eliminate creation of mod_python_so.so in non-windows environments. Fix is ready to commit. Not Done. I decided to defer this for reasons I won't go into just now. It is not a show stopper anyway. Isn't the fix basically just placing the ModPyModule and setup() with ModPyModule inside the if winbuild block and then having another set() without the ModPyModule in the else clause? Unless there is some good reason for it, I think it is a show stopper because it makes the build process look a bit on the bizzare side on Unix. Grisha
Re: Getting ready for 3.2 beta 2
On 01/09/2005, at 6:19 AM, Jim Gallacher wrote:Hey Gang,I think we are ready for the 3.2.1b release. If there are no objections in the next 24 hours I'll create the package and make the announcement on python-dev.Sounds good.I'll always be hoping to sneak in just one more change (eg. MODPYTHON-73),but realities are that I have to stop at some point. :-)BTW, I will be traveling for a few weeks from this weekend and at timeswill be disconnected from the Internet and at other times will only havebasic dial-up access, so you might not hear from me too much duringthat period.Maybe I'll use the time to dream about writing a book. ;-)Graham
Re: Getting ready for 3.2 beta 2
Hi Jim, The fix for MODPYTHON-72 should be easy, unfortunately I'm quite busy right now, since my first daughter was born three days ago... I'll do my best to have a look at it, but if someone feels like doing it, I'll understand. Regards, Nicolas2005/8/26, Jim Gallacher [EMAIL PROTECTED]: I think we should aim for the second beta release in the next couple ofdays. I have a few questions and a list of outstanding issues.Name of tarball: mod_python-3.2.1b.tgz?Also, I assume a new branch called tags/release- 3.2.1-BETA will becreated in subversion, correct?Outstanding issues:1. flexFix is ready to commit pending some feedback on the warning messagegenerated by configure.2. MacOS compile Fixed in svn trunk.3. Eliminate creation of mod_python_so.so in non-windows environments.Fix is ready to commit.4. Fix segfault + memory leaks detailed in: http://issues.apache.org/jira/browse/MODPYTHON-75 http://issues.apache.org/jira/browse/MODPYTHON-60 Boyan's patch detailed in MODPYTHON-75 seems to fix both of these. Fix is ready to commit.5. http://issues.apache.org/jira/browse/MODPYTHON-72Fix still required.6. Publisher bug in 3.2 BETA, detailed by Graham in python-dev message posted 2005-08-21.Fix still required.I haven't looked at the code involved in items 5 and 6, but hopefullythe fixes will be fairly trivial.Regards,Jim
Getting ready for 3.2 beta 2
I think we should aim for the second beta release in the next couple of days. I have a few questions and a list of outstanding issues. Name of tarball: mod_python-3.2.1b.tgz? Also, I assume a new branch called tags/release-3.2.1-BETA will be created in subversion, correct? Outstanding issues: 1. flex Fix is ready to commit pending some feedback on the warning message generated by configure. 2. MacOS compile Fixed in svn trunk. 3. Eliminate creation of mod_python_so.so in non-windows environments. Fix is ready to commit. 4. Fix segfault + memory leaks detailed in: http://issues.apache.org/jira/browse/MODPYTHON-75 http://issues.apache.org/jira/browse/MODPYTHON-60 Boyan's patch detailed in MODPYTHON-75 seems to fix both of these. Fix is ready to commit. 5. http://issues.apache.org/jira/browse/MODPYTHON-72 Fix still required. 6. Publisher bug in 3.2 BETA, detailed by Graham in python-dev message posted 2005-08-21. Fix still required. I haven't looked at the code involved in items 5 and 6, but hopefully the fixes will be fairly trivial. Regards, Jim
Re: Getting ready for 3.2 beta 2
Nicolas Lehuen wrote: Hi Jim, The fix for MODPYTHON-72 http://issues.apache.org/jira/browse/MODPYTHON-72 should be easy, unfortunately I'm quite busy right now, since my first daughter was born three days ago... Congratulations Nicolas! I'll do my best to have a look at it, but if someone feels like doing it, I'll understand. I have nothing planned for this weekend, so don't worry if you don't have the time. I should be able to handle it. Best Regards, Jim
Re: Getting ready for 3.2 beta 2
On Thu, 25 Aug 2005, Jim Gallacher wrote: I think we should aim for the second beta release in the next couple of days. I have a few questions and a list of outstanding issues. Name of tarball: mod_python-3.2.1b.tgz? yep, 3.2.1b Also, I assume a new branch called tags/release-3.2.1-BETA will be created in subversion, correct? yup Regards, Jim Thanks Jim! Grisha