Fwd: mod_python 3.3.1 available for testing
-- Forwarded message -- From: Nicolas Lehuen [EMAIL PROTECTED] Date: 1 févr. 2007 20:50 Subject: Re: mod_python 3.3.1 available for testing To: Jim Gallacher [EMAIL PROTECTED] Hi, I've built the 3.3.1 binaries for Windows, you can download it from : http://nicolas.lehuen.com/download/mod_python/ Here are my +1s following the tests : +1 Microsoft Windows XP SP2, Apache 2.0.59 (mpm_winnt), Python 2.4 +1 Microsoft Windows XP SP2, Apache 2.0.59 (mpm_winnt), Python 2.5 +1 Microsoft Windows XP SP2, Apache 2.2.2 (mpm_winnt), Python 2.4 +1 Microsoft Windows XP SP2, Apache 2.2.2 (mpm_winnt), Python 2.5 Once again, no tests for Python 2.3, sadly I need to have a dedicated (or virtual) PC for this and I'm a bit out of time for now. If anyone wants to have a go with Python 2.3, that would be great. And now, time for a Pastis. Regards, Nicolas 2007/1/29, Jim Gallacher [EMAIL PROTECTED]: The mod_python 3.3.1 tarball is available for testing. Hopefully Nicolas will have a chance to create Windows installers for testing in the next couple of days. There have been no changes in the code since the 3.3.0 beta. Indeed the only thing that has changed is the version strings so we don't need extensive testing this time around - just a few people to check that I haven't done something stupid in creating the tarball. Once we have a handful of +1's I'll call for a vote from the core committers. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.3.1.tgz http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.3.1.md5 Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ make check Or for you Windows folks $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Apache, Apache-mpm, Python, the test output, and suggestions, if any). Please present your test results in the following format: +1 OS version, Apache version (apache mpm), Python Version For example: +1 Linux Debian Sid, Apache 2.0.55 (mpm-worker), Python 2.3.5 Presenting your information in a consistent format will help in tabulating the results. You can include additional information in each section, just don't use extra commas. There is no need to include the mod_python version in this string as that information is available in the email subject. Who knows, one day I may actually write a script to extract this information automatically. :) Thank you for your assistance, Jim Gallacher
Re: Core vote for 3.3.1 stable release
Following my tests on Windows, and knowing that 3.3.1 = 3.3.0b + a version number change, I give my +1 on the release. Regards, Nicolas 2007/2/1, Jim Gallacher [EMAIL PROTECTED]: I think we have sufficiently tested 3.3.1 and it is time for the core group to vote on a release. This vote is only open to the mod_python core group (Grisha, Nicolas, Graham and myself) and is binding. We need at least three +1 votes for the release to proceed. A -1 from any of the four voters will veto the release. And here is my vote: +1 Release 3.3.1 for General Availability Jim
Re: Core vote [Re: mod_python 3.3.0 beta available for testing]
+1 for me too ! Nicolas 2006/12/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: core +1 on releasing it into the wild grisha On Mon, 11 Dec 2006, Jim Gallacher wrote: Test results so far, FYI. How long shall we wait until we kick it up to the next level? - jim +1 FreeBSD 6.1, Apache 2.2.3 (mpm-prefork), Python 2.4.3,1 +1 Linux Debian 3.1 Sarge, Apache 2.0.54 (mpm-prefork), Python 2.3.5 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.3.5 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.4.4 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.5 +1 Linux Fedora Core 5 (i386), Apache 2.2.2 (mpm-prefork), Python 2.4.3 +1 Linux Fedora Core 5 i386, Apache 2.2.2 (mpm-prefork), Python 2.4.3 +1 Linux Fedora Core 6, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Fedora Core 6 i386, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Fedora Core 6 x86_64, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Slackware 10.2, Apache 2.2.3 (mpm-prefork), Python 2.4.1 +1 Linux Ubuntu 6.0.6, Apache 2.0.55 (mpm-worker), Python 2.4.3 +1 Linux Ubuntu 6.10, Apache 2.0.55 (mpm-prefork), Python 2.4.4c1 +1 Mac OS X (Darwin 8.8.1), Apache 2.2.3 (mpm-prefork), Python 2.4.2 +1 Mac OS X (PowerPC), Apache 2.2.1 (mpm-worker), Python 2.3.5 (OS Supplied Version) +1 Mac OS X (PowerPC), Apache 2.0.55 (mpm-worker), Python 2.3.5 (OS Supplied Version) +1 Microsoft Windows XP, Apache 2.0.58 (mpm_winnt), Python 2.5 +1 Microsoft Windows XP, Apache 2.0.58 (mpm_winnt), Python 2.4 +1 Microsoft Windows XP, Apache 2.2.2 (mpm_winnt), Python 2.5 +1 Microsoft Windows XP, Apache 2.2.2 (mpm_winnt), Python 2.4
Re: mod_python 3.3.0-dev-20061109 tests on Win32
Guys,First of all sorry for not intervening in the discussion earlier, I haven't had much time for mod_python development lately (hell, not much time for anything except working). The build procedure quoted by Graham at http://www.modpython.org/pipermail/mod_python/2006-September/022092.html works, but of course doesn't use the VC project files or GUI. You should not have to worry about the location of APR or anything like that, the combination of build_installer.bat and the setup.py scripts takes care of everything.Please tell us in what way this procedure fails on your computer. Again, using the VC GUI and project files is not supported right now. Regards,NicolasJeff, one important thing is that VC project files are outdated and should not be used. The 2006/11/13, Jeff Robbins [EMAIL PROTECTED]:Graham,These instructions are not sufficient.The apache environment I have on windows has include files in apachesr/include but also inapachesrc/srclib/apr/include, apachesrc/srclib/apr-iconv/include, andapachesrc/srclib/apr-util/includeSetting the APACHESRC environmental per the instructions only finds the includes in $APACHESRC/include but not the apr files like apr.h in the errorI posted.In the vcproj file, I had to tell the IDE in some dialog where tofind these include files.Is there some other environmental or is there some copy phase in the build on Linux that gets all the include files into$APACHESRC/include?Where is apr.h on your machine?Thanks,Jeff- Original Message -From: Graham Dumpleton [EMAIL PROTECTED]To: python-dev@httpd.apache.orgSent: Sunday, November 12, 2006 20:18Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 Try follow these instructions:http://www.modpython.org/pipermail/mod_python/2006-September/022092.html If these are correct, they probably should be put in the source code if they aren't already. Graham Jeff Robbins wrote .. re: building on Win32 I tried using setup.py but even once I set APACHESRC it still couldn't find the apr* include directories.I set ext_modules = [PSPModule] alone and it built _psp.pyd no problem! C:\work\mod_python-3.3.0-dev-20061109\distpython setup.py build running build running build_py running build_ext building 'mod_python_so' extension C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox/MD /W3 /GX /DNDEBUG -DWIN32 -DNDEBUG -D_WINDOWS -IC:\work\mod_python-3.3.0-dev -20061109\src\include -IC:\work\httpd-2.2.3\include -IC:\Python24\include -IC:\P ython24\PC /TcC:\work\mod_python-3.3.0-dev-20061109\src\mod_python.c /FoC:\work\ mod_python- 3.3.0-dev-20061109\src\mod_python.obj mod_python.c c:\work\httpd-2.2.3\include\ap_config.h(25) : fatal error C1083: Cannot open inc lude file: 'apr.h': No such file or directory error: command 'C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe' failed with exit status 2 - Original Message - From: Graham Dumpleton [EMAIL PROTECTED] To: Jeff Robbins [EMAIL PROTECTED] Cc: python-dev list python-dev@httpd.apache.org Sent: Saturday, November 11, 2006 20:18 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 On 12/11/2006, at 12:31 AM, Jeff Robbins wrote: 3 problems found on Win32: 1) _psp didn't build and I don't know how to build it How are you trying to build mod_python in the first place? Are you using dist/build_installer.bat or using VisualStudio project file. The latter isn't really used any longer and isn't tested. We know that it doesn't list the finfoobject.c file for a start. 2) In the 'Testing PythonImport' test, the path separators in thetwo paths being compared are different (no doubt due to Win32backslash vs forward slash issues) the tests.py code does this: directory = os.path.dirname(__file__) assert( sys.path.count(directory) == 1) os.path.dirname(__file__) is 'C:\\work\\mod_python-3.3.0- dev-20061109\\test\\htdocs' yet sys.path has this in it 'C:/work/mod_python-3.3.0-dev-20061109/ testhtdocs' so the assert fails since the first string can't be found insys.path (count == 0) If in test/test.py you change: c = Container(PythonPath([r'%s']+sys.path % DOCUMENT_ROOT), to: c = Container(PythonPath([r'%s']+sys.path % os.path.normpath(DOCUMENT_ROOT)), does it pass? 3) in test_interpreter_per_directory() the code does this: rsp = self.vhost_get(test_interpreter_per_directory, '/ subdir/foo.py').upper() interpreter+'SUBDIR/' is 'C:/WORK/MOD_PYTHON- 3.3.0-DEV-20061109/ TEST/HTDOCS/SUBDIR/' rsp is 'C:/WORK/MOD_PYTHON-3.3.0-DEV-20061109/TEST/HTDOCS/' I don't understand the tests.py code but it looks like in the interpreter() code def interpreter(req): if req.phase == PythonFixupHandler: if req.filename[-1] != '/' and os.path.isdir (req.filename): req.write(req.interpreter) return apache.DONE return apache.OK else: req.write(req.interpreter) return apache.DONE perhaps the req.filename 'C:/work/mod_python-3.3.0-dev-20061109/ test/htdocs/subdir' is supposed to pass the
Re: mod_python 3.3.0-dev-20061109 tests on Win32
Indeed, the APACHESRC variable has a slightly misleading name, since it doesn't need the full blown source installation. When building mod_python I'm using a stock Win32 Apache 2.0 or 2.2 binary build downloaded from http://httpd.apache.org/, not a source distribution. It may or may not work with a source distribution, but I'm positive it does with a binary one, so Jeff, you should definitely try it this way.Regards,Nicolas 2006/11/13, Graham Dumpleton [EMAIL PROTECTED]: Jeff Robbins wrote .. Graham, These instructions are not sufficient.The apache environment I have on windows has include files in apachesr/include but also in apachesrc/srclib/apr/include, apachesrc/srclib/apr-iconv/include, and apachesrc/srclib/apr-util/include Setting the APACHESRC environmental per the instructions only finds the includes in $APACHESRC/include but not the apr files like apr.h in the error I posted.In the vcproj file, I had to tell the IDE in some dialog where to find these include files.Is there some other environmental or is there some copy phase in the build on Linux that gets all the include files into $APACHESRC/include?All this suggests you are setting APACHESRC to where the original source codefor Apache resides. Can you see if there is a distinct area where the includefiles are installed into along with Apache binaries, modules, config etc. I'm not a Windows person, but do you have a \Apache2 directory with an includedirectory under that. If so, set APACHESRC to \Apache2. If not, then will haveto hope Nicolas is reading email at the moment and comment and he is the one who normally builds the Win32 binary releases for us. Where is apr.h on your machine?In the single include directory along with ap_*.h header files etc where Apachewas installed into. Graham - Original Message - From: Graham Dumpleton [EMAIL PROTECTED] To: python-dev@httpd.apache.org Sent: Sunday, November 12, 2006 20:18 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 Try follow these instructions: http://www.modpython.org/pipermail/mod_python/2006-September/022092.html If these are correct, they probably should be put in the source code if they aren't already. Graham Jeff Robbins wrote .. re: building on Win32 I tried using setup.py but even once I set APACHESRC it still couldn't find the apr* include directories.I set ext_modules = [PSPModule] alone and it built _psp.pyd no problem! C:\work\mod_python-3.3.0-dev-20061109\distpython setup.py build running build running build_py running build_ext building 'mod_python_so' extension C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -DWIN32 -DNDEBUG -D_WINDOWS -IC:\work\mod_python- 3.3.0-dev -20061109\src\include -IC:\work\httpd-2.2.3\include -IC:\Python24\include -IC:\P ython24\PC /TcC:\work\mod_python-3.3.0-dev-20061109\src\mod_python.c /FoC:\work\ mod_python-3.3.0-dev-20061109\src\mod_python.obj mod_python.c c:\work\httpd-2.2.3\include\ap_config.h(25) : fatal error C1083: Cannot open inc lude file: 'apr.h': No such file or directory error: command 'C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe' failed with exit status 2 - Original Message - From: Graham Dumpleton [EMAIL PROTECTED] To: Jeff Robbins [EMAIL PROTECTED] Cc: python-dev list python-dev@httpd.apache.org Sent: Saturday, November 11, 2006 20:18 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 On 12/11/2006, at 12:31 AM, Jeff Robbins wrote: 3 problems found on Win32: 1) _psp didn't build and I don't know how to build it How are you trying to build mod_python in the first place? Are you using dist/build_installer.bat or using VisualStudio project file. The latter isn't really used any longer and isn't tested. We know that it doesn't list the finfoobject.c file for a start. 2) In the 'Testing PythonImport' test, the path separators in the two paths being compared are different (no doubt due to Win32backslash vs forward slash issues) the tests.py code does this: directory = os.path.dirname(__file__) assert( sys.path.count(directory) == 1) os.path.dirname(__file__) is 'C:\\work\\mod_python-3.3.0- dev-20061109\\test\\htdocs' yet sys.path has this in it 'C:/work/mod_python-3.3.0-dev-20061109/ testhtdocs' so the assert fails since the first string can't be found in sys.path (count == 0) If in test/test.py you change: c = Container(PythonPath([r'%s']+sys.path % DOCUMENT_ROOT), to: c = Container(PythonPath([r'%s']+sys.path % os.path.normpath(DOCUMENT_ROOT)), does it pass? 3) in test_interpreter_per_directory() the code does this: rsp = self.vhost_get(test_interpreter_per_directory, '/ subdir/foo.py').upper() interpreter+'SUBDIR/' is 'C:/WORK/MOD_PYTHON-3.3.0-DEV-20061109/ TEST/HTDOCS/SUBDIR/' rsp is 'C:/WORK/MOD_PYTHON- 3.3.0-DEV-20061109/TEST/HTDOCS/' I don't
Re: mod_python 3.3.0-dev-20061109 tests on Win32
Yeah, the 2.2 version needs to tweak setup.py. We should include some kinf of auto-detection of the 2.2 version and act accordingly.Glad to hear that you've succeeded in building mod_python.Regards,Nicolas 2006/11/13, Jeff Robbins [EMAIL PROTECTED]: Nicolas, I downloaded the stock 2.2.3 binary build. To get setup.py to link, I had to edit this: if winbuild: libraries = ['libhttpd', 'libapr-1', 'libaprutil-1', 'ws2_32'] (added the -1 to libapr and libaprutil) The resultant build produced _psp.pyd and also a mod_python_so.pyd which I renamed mod_python.so and it ran. Does this sound right? - Jeff - Original Message - From: Nicolas Lehuen To: Graham Dumpleton Cc: python-dev@httpd.apache.org Sent: Sunday, November 12, 2006 21:04 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 Indeed, the APACHESRC variable has a slightly misleading name, since it doesn't need the full blown source installation. When building mod_python I'm using a stock Win32 Apache 2.0 or 2.2 binary build downloaded from http://httpd.apache.org/, not a source distribution. It may or may not work with a source distribution, but I'm positive it does with a binary one, so Jeff, you should definitely try it this way.Regards,Nicolas 2006/11/13, Graham Dumpleton [EMAIL PROTECTED]: Jeff Robbins wrote .. Graham, These instructions are not sufficient.The apache environment I have on windows has include files in apachesr/include but also in apachesrc/srclib/apr/include, apachesrc/srclib/apr-iconv/include, and apachesrc/srclib/apr-util/include Setting the APACHESRC environmental per the instructions only finds the includes in $APACHESRC/include but not the apr files like apr.h in the error I posted.In the vcproj file, I had to tell the IDE in some dialog where to find these include files.Is there some other environmental or is there some copy phase in the build on Linux that gets all the include files into $APACHESRC/include?All this suggests you are setting APACHESRC to where the original source codefor Apache resides. Can you see if there is a distinct area where the includefiles are installed into along with Apache binaries, modules, config etc. I'm not a Windows person, but do you have a \Apache2 directory with an includedirectory under that. If so, set APACHESRC to \Apache2. If not, then will haveto hope Nicolas is reading email at the moment and comment and he is the one who normally builds the Win32 binary releases for us. Where is apr.h on your machine?In the single include directory along with ap_*.h header files etc where Apachewas installed into.Graham - Original Message - From: Graham Dumpleton [EMAIL PROTECTED] To: python-dev@httpd.apache.org Sent: Sunday, November 12, 2006 20:18 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 Try follow these instructions: http://www.modpython.org/pipermail/mod_python/2006-September/022092.html If these are correct, they probably should be put in the source code if they aren't already. Graham Jeff Robbins wrote .. re: building on Win32 I tried using setup.py but even once I set APACHESRC it still couldn't find the apr* include directories.I set ext_modules = [PSPModule] alone and it built _psp.pyd no problem! C:\work\mod_python-3.3.0-dev-20061109\distpython setup.py build running build running build_py running build_ext building 'mod_python_so' extension C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -DWIN32 -DNDEBUG -D_WINDOWS -IC:\work\mod_python- 3.3.0-dev -20061109\src\include -IC:\work\httpd-2.2.3\include -IC:\Python24\include -IC:\P ython24\PC /TcC:\work\mod_python-3.3.0-dev-20061109\src\mod_python.c /FoC:\work\ mod_python-3.3.0-dev-20061109\src\mod_python.obj mod_python.c c:\work\httpd-2.2.3\include\ap_config.h(25) : fatal error C1083: Cannot open inc lude file: 'apr.h': No such file or directory error: command 'C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe' failed with exit status 2 - Original Message - From: Graham Dumpleton [EMAIL PROTECTED] To: Jeff Robbins [EMAIL PROTECTED] Cc: python-dev list python-dev@httpd.apache.org Sent: Saturday, November 11, 2006 20:18 Subject: Re: mod_python 3.3.0-dev-20061109 tests on Win32 On 12/11/2006, at 12:31 AM, Jeff Robbins wrote: 3 problems found on Win32: 1) _psp didn't build and I don't know how to build it How are you trying to build mod_python in the first place? Are you using dist/build_installer.bat or using VisualStudio project file
mod_python 3.2.10 binaries for Win32
Hi,Here are the Win32 binaries for mod_python 3.2.10 :http://nicolas.lehuen.com/download/mod_python/There is one version for Python 2.3 + Apache 2.0, a version for Python 2.4 + Apache 2.0 and a version for Python 2.4 + Apache 2.2.All three version have passed the unit tests, so we can release them as soon as 3.2.10 is officially released.Regards, Nicolas
Re: Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
OK, I'm currently checking in the fixes you suggested on the trunk. Too bad we cannot write a unit test that checks for memory leaks.Jim, Graham, what shall we do for the 3.2.9 release ? Shall we keep on with the current branch or backport the fixes ? Regards,Nicolas2006/7/9, Harold Ship [EMAIL PROTECTED]: I've made a test build based on 3.2.8 release, where I've added Py_XDECREF()calls in parse_qsl(), cfgtree_walk() TWICE (one on t, one on child), andreq_readlines().My foo/bar program doesn't leak, and I'm now testing my full application. So far, it seems to be ok.Harold
[jira] Resolved: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
[ http://issues.apache.org/jira/browse/MODPYTHON-172?page=all ] Nicolas Lehuen resolved MODPYTHON-172: -- Fix Version: 3.3 Resolution: Fixed Fixed in the trunk, we need to apply changes from revision #420275 if we want to backport this into the 3.2 branch. Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55 -- Key: MODPYTHON-172 URL: http://issues.apache.org/jira/browse/MODPYTHON-172 Project: mod_python Type: Bug Components: core Versions: 3.2.8 Environment: Win32 XP SP1 / SP2 Apache 2.0.55 installed from binary (.MSI) Python 2.4.2 or 2.4.3installed from binary from www.python.org Reporter: Laurent Blanquet Fix For: 3.3 I encounter memory leaks [~ 16 K per request) using the configuration described below. = Python configuration from Httpd.conf: = Alias /python/ d:/python24/B2B/ Directory d:/python24/B2B AddHandler mod_python .py PythonHandler pyHandlerHTTP PythonDebug On /Directory = Test handler - pyHandlerHTTP.py : = import mod_python from mod_python import util def handler(req): #Removing this line solves the problem. F=util.FieldStorage( req ) return mod_python.apache.OK = HTTP Request (dump using TCPWATCH): = POST http://localhost:80/python/Alertes.py HTTP/1.0 Content-Type: multipart/form-data; boundary=061006144341906 Content-Length: 209 Proxy-Connection: keep-alive Host: www.tx2-localhost Accept: text/html, */* User-Agent: Mozilla/3.0 (compatible; Indy Library) Proxy-Authorization: Basic Og== --061006144341906 Content-Disposition: form-data; name=TYPE LAST_ALERTS --061006144341906 Content-Disposition: form-data; name=FILEAGE 180 --061006144341906 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: *** PROBABLY SPAM *** RE: Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
Yes, I've done the same except that I've used Py_DECREF since the references are guarded by previous non null tests.https://svn.apache.org/repos/asf/httpd/mod_python/trunk/src/util.c
Regards,Nicolas2006/7/9, Harold J. Ship [EMAIL PROTECTED]:
Just
to be sure, this is the change I've made to util.c (marked with
@HJS):
while (dir) {
PyObject *t =
Py_BuildValue((s, s), dir-directive,
dir-args); if
(!t)
return PyErr_NoMemory();
PyList_Append(list,
t); Py_XDECREF(t); //
@HJS
if (dir-first_child)
{
PyObject *child =
cfgtree_walk(dir-first_child);
if
(!child)
return PyErr_NoMemory();
PyList_Append(list,
child);
Py_XDECREF(child); // @HJS
}
dir =
dir-next; }
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] On Behalf Of Nicolas
LehuenSent: Sunday, July 09, 2006 11:46 AMTo: Harold J.
ShipCc: python-dev@httpd.apache.orgSubject: Re: Commented:
(MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on
apache 2.0.55
OK, I'm currently checking in the fixes you suggested on the trunk.
Too bad we cannot write a unit test that checks for memory leaks.Jim,
Graham, what shall we do for the 3.2.9 release ? Shall we keep on with the
current branch or backport the fixes ? Regards,Nicolas
2006/7/9, Harold Ship [EMAIL PROTECTED]:
I've
made a test build based on 3.2.8 release, where I've added
Py_XDECREF()calls in parse_qsl(), cfgtree_walk() TWICE (one on t, one on
child), andreq_readlines().My foo/bar program doesn't leak, and
I'm now testing my full application. So far, it seems to be
ok.Harold
Re: svn commit: r420275 - in /httpd/mod_python/trunk/src: _apachemodule.c requestobject.c util.c
Yeah, I forgot about the appendix C in the documentation. I'm going to correct this ASAP.I know about the sizehint problem, I'm currently working on it. I just wanted to fix it in a different commit to make backporting more easy. Also, I want to write a unit test for it. This should be done in a few hours, since I'm still on a holiday schedule :).
Regards,Nicolas2006/7/9, Graham Dumpleton [EMAIL PROTECTED]:
On 09/07/2006, at 8:05 PM, [EMAIL PROTECTED] wrote: Modified: httpd/mod_python/trunk/src/requestobject.c URL:
http://svn.apache.org/viewvc/httpd/mod_python/trunk/src/ requestobject.c?rev=420275r1=420274r2=420275view=diff ==
--- httpd/mod_python/trunk/src/requestobject.c (original) +++ httpd/mod_python/trunk/src/requestobject.c Sun Jul9 03:05:30 2006 @@ -1139,6 +1139,7 @@PyObject *line, *rlargs;
long sizehint = -1;long size = 0; +long linesize;if (! PyArg_ParseTuple(args, |l, sizehint))return NULL; @@ -1151,13 +1152,15 @@
return PyErr_NoMemory();line = req_readline(self, rlargs); -while (line (PyString_Size(line)0)) { +while (line ((linesize=PyString_Size(line))0)) {
PyList_Append(result, line); -size += PyString_Size(line); +size += linesize;if ((sizehint != -1) (size = size))break;
+Py_DECREF(line);line = req_readline(self, args);} +Py_XDECREF(line);if (!line)return NULL;Did you forget to commit Doc/appendixc.tex bug fix list and version
files increment changes?BTW, we still need to address: http://issues.apache.org/jira/browse/MODPYTHON-179in that function. :-)
Sorry for not helping much of late. Have got myself side tracked onmy own software.Graham
[jira] Resolved: (MODPYTHON-179) req.readlines(sizehint) does not work correctly
[ http://issues.apache.org/jira/browse/MODPYTHON-179?page=all ]
Nicolas Lehuen resolved MODPYTHON-179:
--
Fix Version: 3.3
Resolution: Fixed
Fixed and added two unit tests on the trunk. Apply revision #420288 if this
needs to be backported on the 3.2 branch.
req.readlines(sizehint) does not work correctly
---
Key: MODPYTHON-179
URL: http://issues.apache.org/jira/browse/MODPYTHON-179
Project: mod_python
Type: Bug
Components: core
Versions: 3.2.8
Environment: All
Reporter: Jim Gallacher
Assignee: Jim Gallacher
Priority: Minor
Fix For: 3.3
A bug in req_readlines(sizehint) in requestobject.c causes output to be
returned prematurely for any value of the optional sizehint argument.
The faulty bit of code is:
line = req_readline(self, rlargs);
while (line (PyString_Size(line)0)) {
PyList_Append(result, line);
size += PyString_Size(line);
if ((sizehint != -1) (size = size))
break;
line = req_readline(self, args);
}
Since (size = size) will always be true, reading the input stream will end
prematurely for any value of sizehint != -1.
This code will fix the problem.
--- requestobject.c (revision 417294)
+++ requestobject.c (working copy)
@@ -1154,7 +1154,7 @@
while (line (PyString_Size(line)0)) {
PyList_Append(result, line);
size += PyString_Size(line);
-if ((sizehint != -1) (size = size))
+if ((sizehint != -1) (size = sizehint))
break;
line = req_readline(self, args);
}
Once that is fixed the documentation needs to be revised, as it does not
accurately reflect the behaviour of the code.
Currently the docs are:
Reads all or up to sizehint bytes of lines using readline and returns a
list of the lines read.
whereas the code can read beyond sizehint. The total read could actually be
sizehint + len(line) where line is the last line read.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
[ http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12419906 ] Nicolas Lehuen commented on MODPYTHON-172: -- I've just backported the fix into the 3.2 branch. Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55 -- Key: MODPYTHON-172 URL: http://issues.apache.org/jira/browse/MODPYTHON-172 Project: mod_python Type: Bug Components: core Versions: 3.2.8 Environment: Win32 XP SP1 / SP2 Apache 2.0.55 installed from binary (.MSI) Python 2.4.2 or 2.4.3installed from binary from www.python.org Reporter: Laurent Blanquet Fix For: 3.3 I encounter memory leaks [~ 16 K per request) using the configuration described below. = Python configuration from Httpd.conf: = Alias /python/ d:/python24/B2B/ Directory d:/python24/B2B AddHandler mod_python .py PythonHandler pyHandlerHTTP PythonDebug On /Directory = Test handler - pyHandlerHTTP.py : = import mod_python from mod_python import util def handler(req): #Removing this line solves the problem. F=util.FieldStorage( req ) return mod_python.apache.OK = HTTP Request (dump using TCPWATCH): = POST http://localhost:80/python/Alertes.py HTTP/1.0 Content-Type: multipart/form-data; boundary=061006144341906 Content-Length: 209 Proxy-Connection: keep-alive Host: www.tx2-localhost Accept: text/html, */* User-Agent: Mozilla/3.0 (compatible; Indy Library) Proxy-Authorization: Basic Og== --061006144341906 Content-Disposition: form-data; name=TYPE LAST_ALERTS --061006144341906 Content-Disposition: form-data; name=FILEAGE 180 --061006144341906 -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
Hi Harold,With Visual Studio .NET 2003, it's quite easy, just cd into the dist directory and launch build_installer.bat. You should eventually get an installer into the dist/dist directory. Note that with Apache 2.2 , you may need to tweak the setup.py.in file manually a little bit.Regards,Nicolas2006/7/8, Harold J. Ship [EMAIL PROTECTED]:Thanks all.I wasn't worried. I would also like to help, if possible. Can someone point out some instructions on building mod_python on Windows? I'm usingActivestate Python 2.3.5, and have Visual Studio 6.0 and .NET 2003.Harold
Re: [jira] Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
Yeah Harold, thanks for the time you are spending on this bug.I'm not ignoring you either, I'm just on vacation, trying to disconnect from my work in particular and the net in general. Not so easy as this mail tends to show. I'll have a look at this too when I come back to civilization.Regards,Nicolas2006/7/7, Jim Gallacher [EMAIL PROTECTED] :Hi Harold,I just wanted to let you know you are not being ignored. I just need some free time to dive into this - hopefully this weekend.JimHarold Ship (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12419728 ] Harold Ship commented on MODPYTHON-172: --- Please look at lines 202-205 of the same file, in parse_qs() : PyObject *list; list = Py_BuildValue([O], val); PyDict_SetItem(dict, key, list); Py_DECREF(list); Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55 --Key: MODPYTHON-172URL: http://issues.apache.org/jira/browse/MODPYTHON-172Project: mod_python Type: Bug Components: core Versions: 3.2.8Environment: Win32 XPSP1 / SP2 Apache 2.0.55installed from binary (.MSI) Python 2.4.2or2.4.3installed from binary from www.python.org Reporter: Laurent Blanquet I encounter memory leaks [~ 16 K per request) using the configuration described below. = Python configuration from Httpd.conf: = Alias /python/ d:/python24/B2B/ Directory d:/python24/B2B AddHandler mod_python .py PythonHandler pyHandlerHTTP PythonDebug On /Directory = Test handler -pyHandlerHTTP.py : = import mod_python from mod_python import util def handler(req): #Removing this line solves the problem. F=util.FieldStorage( req ) return mod_python.apache.OK = HTTP Request (dump using TCPWATCH): = POST http://localhost:80/python/Alertes.py HTTP/1.0 Content-Type: multipart/form-data; boundary=061006144341906 Content-Length: 209 Proxy-Connection: keep-alive Host: www.tx2-localhost Accept: text/html, */* User-Agent: Mozilla/3.0 (compatible; Indy Library) Proxy-Authorization: Basic Og== --061006144341906 Content-Disposition: form-data; name=TYPE LAST_ALERTS --061006144341906 Content-Disposition: form-data; name=FILEAGE 180 --061006144341906
Re: mod_python 3.2.9 available for testing
I've got :+1 Windows XP SP2, Apache 2.0.58 (mpm_winnt), Python 2.4.3But, and it's my fault for not having tested this for a long time :-1 Windows XP SP2, Apache 2.2.2 (mpm_winnt), Python 2.4.3 Only two tests fail but with a segfault, it's test_srv_register_cleanup and test_apache_register_cleanup. This is not really surprising... I think we should go ahead and release the 3.2.9 version, while filing a known bug regarding the fact that we drop the support for those two functions. If we accept this, then it's a +1. Regards,Nicolas2006/6/30, Jeff Hinrichs - DMT [EMAIL PROTECTED]: +1 FreeBSD 6.1-RELEASE-p2, Apache 2.2 (mpm-prefork), Python 2.4.3On 6/29/06, Jim Gallacher [EMAIL PROTECTED] wrote: +1 Linux Ubuntu 6.06 Dapper Drake, Apache 2.0.55 (mpm-worker), Python 2.4.3 Jim Gallacher wrote: The mod_python 3.2.9 tarball is available for testing. This tarball is unchanged from 3.2.9-rc3, but should be retested anyway - just in case something went pair-shaped in the process of tagging and packaging. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://people.apache.org/~jgallacher/mod_python/dist/ http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9.tgz http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9.tgz.md5 Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Apache, Apache-mpm, Python, the test output, and suggestions, if any). Please present your test results in the following format: +1 OS version, Apache version (apache mpm), Python Version For example: +1 Linux Debian Sid, Apache 2.0.55 (mpm-worker), Python 2.3.5 Presenting your information in a consistent format will help in tabulating the results. You can include additional information in each section, just don't use extra commas. There is no need to include the mod_python version in this string as that information is available in the email subject. Who knows, one day I may actually write a script to extract this information automatically. :) Thank you for your assistance, Jim Gallacher --Jeff HinrichsDundee Media Technology, Inc [EMAIL PROTECTED]402.320.0821
Re: [jira] Commented: (MODPYTHON-172) Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55
Hi,
The subject of this thread is about mod_python 3.2.8, yet you report
using mod_python 3.1.3. Which version have you detected the memory leak
in ? There are a bunch of leaks that have been fixed since 3.1.3...Regards,Nicolas2006/6/30, Harold Ship (JIRA) [EMAIL PROTECTED]
:[
http://issues.apache.org/jira/browse/MODPYTHON-172?page=comments#action_12418577 ]Harold Ship commented on MODPYTHON-172:---I've been able to reproduce the problem with mod_python.publisher.
Using:Windows 2000 Professionalapache 2.0.54python 2.3.5mod_python 3.1.3the following script:#foo.pydef bar(req):a=req.form.getfirst('a')req.write('a=%s'%str(a))
req.write('Ok')repeatedly sending requests to addr/foo/bar?a=b causes memory to rise continuously. removing the querystring does not. I used task manager to measure memory.I've also found that the following change to
util.py corrects this behaviour:old util.py:parse_qs = _apache.parse_qsparse_qsl = _apache.parse_qslworkaround util.py:parse_qs = _apache.parse_qs#parse_qsl = _apache.parse_qslfrom cgi import parse_qsl
Memory leak with util.fieldstorage using mod_python 3.2.8 on apache 2.0.55 --Key: MODPYTHON-172
URL: http://issues.apache.org/jira/browse/MODPYTHON-172Project: mod_python Type: Bug Components: core
Versions: 3.2.8Environment: Win32 XPSP1 / SP2 Apache 2.0.55installed from binary (.MSI) Python 2.4.2or2.4.3installed from binary from www.python.org
Reporter: Laurent Blanquet I encounter memory leaks [~ 16 K per request) using the configuration described below. = Python configuration from
Httpd.conf: = Alias /python/ d:/python24/B2B/ Directory d:/python24/B2B AddHandler mod_python .py PythonHandler pyHandlerHTTP
PythonDebug On /Directory = Test handler -pyHandlerHTTP.py : = import mod_python from mod_python import util
def handler(req): #Removing this line solves the problem. F=util.FieldStorage( req ) return mod_python.apache.OK = HTTP Request (dump using TCPWATCH):
= POST http://localhost:80/python/Alertes.py HTTP/1.0 Content-Type: multipart/form-data; boundary=061006144341906
Content-Length: 209 Proxy-Connection: keep-alive Host: www.tx2-localhost Accept: text/html, */* User-Agent: Mozilla/3.0 (compatible; Indy Library) Proxy-Authorization: Basic Og==
--061006144341906 Content-Disposition: form-data; name=TYPE LAST_ALERTS --061006144341906 Content-Disposition: form-data; name=FILEAGE
180 --061006144341906--This message is automatically generated by JIRA.-If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa-For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: mod_python 3.2.9-rc3 available for testing
+1 Windows XP SP2, Apache 2.0.58, ActivePython 2.4.3, mod_python 3.2.9-rc3 Nicolas 2006/6/25, Jim Gallacher [EMAIL PROTECTED]: The mod_python 3.2.9-rc3 tarball is available for testing. This release adds support for apache 2.2 as well as some other useful backports from the development branch. For information on the changes from 3.2.8 take a look at doc-html/node98.html in the tarball. The only difference from 3.2.9-rc2 is that the FieldStorage modifications (MODPYTHON-93) that were causing problems for some applications such as Trac have been reverted. FieldStorage behaviour should now be the same as in 3.2.8. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://people.apache.org/~jgallacher/mod_python/dist/ http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc3.tgz http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc3.tgz.md5 Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). If this tarball looks good, I'll tag it svn and create a 3.2.9 final tarball. Thank you for your assistance, Jim Gallacher
Re: mod_python 3.2.9-rc2 available for testing
+1 Windows XP SP2, ActivePython 2.4.3, Apache 2.0.58 Regards, Nicolas 2006/6/23, Jim Gallacher [EMAIL PROTECTED]: OK, this time for real. :) The mod_python 3.2.9-rc2 tarball is available for testing. This release adds support for apache 2.2 as well as some other useful backports from the development branch. For information on the changes from 3.2.8 take a look at doc-html/node98.html in the tarball. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://people.apache.org/~jgallacher/mod_python/dist/ http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc1.tgz http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc1.tgz.md5 Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). If this tarball looks good, I'll tag it svn and create a 3.2.9 final tarball. Thank you for your assistance, Jim Gallacher
Re: 3.2.9-rc2 FieldStorage Problems (was Re: mod_python 3.2.9-rc2 available for testing)
* How are applications supposed to perform write operations on a FieldStorage, in 3.3 and the future? Personally I never considered writing to FieldStorage. I always thought of it as a read-only representation of a submitted form, but then that's just my mental map. It's a pretty uncommon usage - it surprised me when I saw it in Trac. Trac is using it to bundle additional request information gathered from sources other than the form itself - it makes me suspect that someone in the past thought hey, we have this dictionary-like thing we are passing around already - wouldn't it be nice if we could shove extra things in it, rather than passing around another object too?, and then proceeded to hack things so it was possible. What I don't understand is why they don't put extra attributes in the request object itself, since it supports that. This is done everywhere in the publisher and the PSP handler. Fiddling with the FieldStorage looks messy to me too - then again, if we promise a dict-like interface we should make sure it is supported. Regards, Nicolas
Re: SSI crashes on Win32.
Hi Graham, After a few interesting weeks, I finally manage to get some time to help on mod_python. I've ran the tests on the latest Subversion revision (407968) with Apache 2.0.58 and Python 2.4.3, and all test pass, including the SSI ones. For a yet unknown reason, every test fail with a debug build of the Python trunk with the aformentioned Invalid thread state for this thread error. I don't know whether this is due to a mistake from me, a bug in the Python trunk, or if it is that the debug build is a little bit more cautious about thread state management and breaks earlier than the non-debug version when faced by a faulty mod_python thread state management code... I'll try to see if we still have the segfault that could be sometimes observed when the Apache server is stopped, as it may be related. Best regards, Nicolas 2006/5/9, Graham Dumpleton [EMAIL PROTECTED]: Nicolas Lehuen wrote .. Hi Graham, The latest trunk version yields multiple segfaults and failures in different places : * Testing req.add_handler() for empty phase E * Testing req.add_handler() directory E * Testing interpreter per directive E * Testing phase status F * Testing server side include F Okay Nicolas, can you check out latest and give it a go again. Got rid of obvious mistake of not deleting old variable definition. My checking was not as rigourous as it should as I should have picked up that interpreter name was wrong. The phase status example will probably still fail as don't understand that one yet. It may be an auth setup issue in test suite configuration for that specific test. Graham
Re: SSI crashes on Win32.
I've forgot to mention the platform : as usual for me, it's Windows XP SP2. Regards, Nicolas 2006/5/20, Nicolas Lehuen [EMAIL PROTECTED]: Hi Graham, After a few interesting weeks, I finally manage to get some time to help on mod_python. I've ran the tests on the latest Subversion revision (407968) with Apache 2.0.58 and Python 2.4.3, and all test pass, including the SSI ones. For a yet unknown reason, every test fail with a debug build of the Python trunk with the aformentioned Invalid thread state for this thread error. I don't know whether this is due to a mistake from me, a bug in the Python trunk, or if it is that the debug build is a little bit more cautious about thread state management and breaks earlier than the non-debug version when faced by a faulty mod_python thread state management code... I'll try to see if we still have the segfault that could be sometimes observed when the Apache server is stopped, as it may be related. Best regards, Nicolas 2006/5/9, Graham Dumpleton [EMAIL PROTECTED]: Nicolas Lehuen wrote .. Hi Graham, The latest trunk version yields multiple segfaults and failures in different places : * Testing req.add_handler() for empty phase E * Testing req.add_handler() directory E * Testing interpreter per directive E * Testing phase status F * Testing server side include F Okay Nicolas, can you check out latest and give it a go again. Got rid of obvious mistake of not deleting old variable definition. My checking was not as rigourous as it should as I should have picked up that interpreter name was wrong. The phase status example will probably still fail as don't understand that one yet. It may be an auth setup issue in test suite configuration for that specific test. Graham
Re: SSI crashes on Win32.
OK, it seems that my last hypothesis was the good one : mod_python is
doing things with thread states that are frowned upon by debug build.
The tests are only performed in the debug build, so that's why we had
no problem with the release build.
The tests are found in pystate.c, in function PyThreadState_Swap (line
297 in the current trunk revision) :
/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in debug
builds.
*/
#if defined(Py_DEBUG) defined(WITH_THREAD)
if (newts) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check check-interp == newts-interp check != newts)
Py_FatalError(Invalid thread state for this thread);
}
#endif
So it seems mod_python is trying to share the same thread state
between multiple threads, which should not be possible. If someone
has an idea, please help me, because I'm not really up to date about
thread state management, so I'll have to read a fair bit of
documentation before understanding what's happening there.
Regards,
Nicolas
2006/5/20, Nicolas Lehuen [EMAIL PROTECTED]:
I've forgot to mention the platform : as usual for me, it's Windows XP SP2.
Regards,
Nicolas
2006/5/20, Nicolas Lehuen [EMAIL PROTECTED]:
Hi Graham,
After a few interesting weeks, I finally manage to get some time to
help on mod_python.
I've ran the tests on the latest Subversion revision (407968) with
Apache 2.0.58 and Python 2.4.3, and all test pass, including the SSI
ones.
For a yet unknown reason, every test fail with a debug build of the
Python trunk with the aformentioned Invalid thread state for this
thread error.
I don't know whether this is due to a mistake from me, a bug in the
Python trunk, or if it is that the debug build is a little bit more
cautious about thread state management and breaks earlier than the
non-debug version when faced by a faulty mod_python thread state
management code... I'll try to see if we still have the segfault that
could be sometimes observed when the Apache server is stopped, as it
may be related.
Best regards,
Nicolas
2006/5/9, Graham Dumpleton [EMAIL PROTECTED]:
Nicolas Lehuen wrote ..
Hi Graham,
The latest trunk version yields multiple segfaults and failures in
different places :
* Testing req.add_handler() for empty phase
E
* Testing req.add_handler() directory
E
* Testing interpreter per directive
E
* Testing phase status
F
* Testing server side include
F
Okay Nicolas, can you check out latest and give it a go again. Got rid of
obvious mistake of not deleting old variable definition. My checking was
not as rigourous as it should as I should have picked up that interpreter
name was wrong.
The phase status example will probably still fail as don't understand that
one yet. It may be an auth setup issue in test suite configuration for that
specific test.
Graham
Python debug build complains about thread state (was Re: SSI crashes on Win32.)
Graham,
In fact PyEval_AcquireThread does call PyThreadState_Swap :
void
PyEval_AcquireThread(PyThreadState *tstate)
{
if (tstate == NULL)
Py_FatalError(PyEval_AcquireThread: NULL new thread state);
/* Check someone has called PyEval_InitThreads() to create the lock */
assert(interpreter_lock);
PyThread_acquire_lock(interpreter_lock, 1);
if (PyThreadState_Swap(tstate) != NULL)
Py_FatalError(
PyEval_AcquireThread: non-NULL old thread state);
}
The error is effectively raised in the WITH_THREAD block of the
get_interpreter function :
/* create thread state and acquire lock */
tstate = PyThreadState_New(idata-istate);
#ifdef WITH_THREAD
PyEval_AcquireThread(tstate);
#else
PyThreadState_Swap(tstate);
#endif
It looks from the Python source code that this kind of way to build a
new thread state with a shared interpreter should not be possible...
I'm not sure I understand evrything here.
BTW, this is not related to SSI or specific to Win32, so I've changed
the subject of this thread.
Regards,
Nicolas
2006/5/20, Graham Dumpleton [EMAIL PROTECTED]:
On 20/05/2006, at 6:27 PM, Nicolas Lehuen wrote:
OK, it seems that my last hypothesis was the good one : mod_python is
doing things with thread states that are frowned upon by debug build.
The tests are only performed in the debug build, so that's why we had
no problem with the release build.
The tests are found in pystate.c, in function PyThreadState_Swap (line
297 in the current trunk revision) :
/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in debug
builds.
*/
#if defined(Py_DEBUG) defined(WITH_THREAD)
if (newts) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check check-interp == newts-interp check != newts)
Py_FatalError(Invalid thread state for this thread);
}
#endif
So it seems mod_python is trying to share the same thread state
between multiple threads, which should not be possible. If someone
has an idea, please help me, because I'm not really up to date about
thread state management, so I'll have to read a fair bit of
documentation before understanding what's happening there.
For each request a new thread state is created.
That is, in get_interpreter() it does:
/* create thread state and acquire lock */
tstate = PyThreadState_New(idata-istate);
#ifdef WITH_THREAD
PyEval_AcquireThread(tstate);
#else
PyThreadState_Swap(tstate);
#endif
In this case, PyThreadState_Swap() isn't called when HAVE_THREAD is
defined, which is only time your check is done.
The problem seems to related to where PyThreadState_Swap() function is
always called in make_interpreter() and PythonChildInitHandler().
Will have to track down what that call is doing at those points.
Maybe those
calls should be:
#ifdef WITH_THREAD
PyEval_ReleaseThread(tstate);
#else
PyThreadState_Swap(NULL);
#endif
Graham
Regards,
Nicolas
2006/5/20, Nicolas Lehuen [EMAIL PROTECTED]:
I've forgot to mention the platform : as usual for me, it's
Windows XP SP2.
Regards,
Nicolas
2006/5/20, Nicolas Lehuen [EMAIL PROTECTED]:
Hi Graham,
After a few interesting weeks, I finally manage to get some time to
help on mod_python.
I've ran the tests on the latest Subversion revision (407968) with
Apache 2.0.58 and Python 2.4.3, and all test pass, including the
SSI
ones.
For a yet unknown reason, every test fail with a debug build of the
Python trunk with the aformentioned Invalid thread state for this
thread error.
I don't know whether this is due to a mistake from me, a bug in the
Python trunk, or if it is that the debug build is a little bit more
cautious about thread state management and breaks earlier than the
non-debug version when faced by a faulty mod_python thread state
management code... I'll try to see if we still have the segfault
that
could be sometimes observed when the Apache server is stopped,
as it
may be related.
Best regards,
Nicolas
2006/5/9, Graham Dumpleton [EMAIL PROTECTED]:
Nicolas Lehuen wrote ..
Hi Graham,
The latest trunk version yields multiple segfaults and
failures in
different places :
* Testing req.add_handler() for empty phase
E
* Testing req.add_handler() directory
E
* Testing interpreter per directive
E
* Testing phase status
F
* Testing server side include
F
Okay Nicolas, can you check out latest and give it a go again.
Got rid of
obvious mistake of not deleting old variable definition. My
checking was
not as rigourous as it should as I should have picked up that
interpreter
name was wrong.
The phase status example will probably still fail as don't
understand
Re: Web archives up...
Thank you very much, Justin !Regards,Nicolas2006/4/20, Justin Erenkrantz [EMAIL PROTECTED]: http://mail-archives.apache.org/mod_mbox/httpd-python-dev/I'm not sure how this slipped through the cracks, but we missed thepython-dev@ and python-cvs@ list in our collection of web archives for apache.org.It's fixed now with the full archives for both lists up now.-- justin
Re: Form of req.filename/req.hlist.directory on Win32 systems.
Hi Graham,Here is the test handler I've used :from mod_python import apachedef handler(req): req.content_type = 'text/plain' req.write(req.hlist.directory+'\n') req.write(req.filename+'\n' ) return apache.OKIf I use :DocumentRoot c:\\apache22\\htdocsDirectory c:\\apache22\\htdocs # ... SetHandler mod_python PythonHandler test_handler /DirectoryI get, when calling http://localhost/index.html:c:\apache22\htdocs/C:/apache22/htdocs/index.htmlNote that the drive letter has been uppercased and req.filename normalized to POSIX path names. req.hlist.directory, though supported by Win32, looks weird.Now with :DocumentRoot c:/apache22/htdocs Directory c:/apache22/htdocs # ... SetHandler mod_python PythonHandler test_handler /Directory I get :c:/apache22/htdocs/C:/apache22/htdocs/index.htmlWith :DocumentRoot c:/apache22/htdocs Directory c:\\apache22\\htdocs # ... SetHandler mod_python PythonHandler test_handler /Directory I get :c:\apache22\htdocs/C:/apache22/htdocs/index.htmlAnd finally with :DocumentRoot c:\\apache22\\htdocs Directory c:/apache22/htdocs # ... SetHandler mod_python PythonHandler test_handler /Directory I get :c:/apache22/htdocs/C:/apache22/htdocs/index.htmlSo req.filename seems always normalized while req.hlist.directory reflects what was entered in the Directory tag. Both POSIX and Windows forms are allowed, unfortunately, but the backslash forms needs C-style escaping, and IIRC the Apache documentation recommends using forward slashes. Regards,Nicolas2006/4/16, Graham Dumpleton [EMAIL PROTECTED]: I am sure I asked this a long time ago, but have forgotten all thedetails.On Win32 systems does req.filename set by Apache always use POSIXstyle forward slashes, ie., '/', to separate components of adirectory? Thus: /some/pathHow does Apache indicate a drive letter when one is necessary? Is it: c:/some/pathDoes any of the above change based on whether forward or backwardslashes are used in a Directory directive? Ie., Directory c:/some/path ... /Directory?vs: Directory c:\\some\\path ... /DirectoryOr does Apache not allow the latter anyway? If Apache does allow the latter, does that mean that req.hlist.directoryis coming through set including backslashes rather than forwardslashes.I want to get my head around this all again as at different times the valuesof req.filename and req.hlist.directory are used to determine the Pythoninterpreter name. As highlighted in: http://issues.apache.org/jira/browse/MODPYTHON-161 If there is a mix of conventions, with user code also being able toaffectthese values, there may be no consistency and thus could end up withscenarios where a different interpreter to one than was expected will be used.Any help from Win32 users in understanding all this would be muchappreciated.Thanks.Graham
Re: Form of req.filename/req.hlist.directory on Win32 systems.
This was with the Subversion trunk.Ill do some tests with 3.2.8 and tell you the results.Nicolas2006/4/17, Graham Dumpleton [EMAIL PROTECTED]:Was this with mod_python from subversion or 3.2.8? Want to qualify whether latest set of changes I checked in to supportFiles directive has caused it to behave differently as how it determinesreq.hlist.directory is different to before.Thanks.Graham On 18/04/2006, at 4:33 AM, Nicolas Lehuen wrote: Hi Graham, Here is the test handler I've used : from mod_python import apache def handler(req): req.content_type = 'text/plain' req.write(req.hlist.directory+'\n') req.write(req.filename+'\n' ) return apache.OK If I use : DocumentRoot c:\\apache22\\htdocs Directory c:\\apache22\\htdocs# ... SetHandler mod_python PythonHandler test_handler /Directory I get, when calling http://localhost/index.html: c:\apache22\htdocs/ C:/apache22/htdocs/index.htmlNote that the drive letter has been uppercased and req.filename normalized to POSIX path names. req.hlist.directory , though supported by Win32, looks weird. Now with : DocumentRoot c:/apache22/htdocs Directory c:/apache22/htdocs# ... SetHandler mod_python PythonHandler test_handler /Directory I get : c:/apache22/htdocs/ C:/apache22/htdocs/index.html With : DocumentRoot c:/apache22/htdocs Directory c:\\apache22\\htdocs# ... SetHandler mod_python PythonHandler test_handler /Directory I get : c:\apache22\htdocs/ C:/apache22/htdocs/index.html And finally with : DocumentRoot c:\\apache22\\htdocs Directory c:/apache22/htdocs# ... SetHandler mod_python PythonHandler test_handler /Directory I get : c:/apache22/htdocs/ C:/apache22/htdocs/index.html So req.filename seems always normalized while req.hlist.directory reflects what was entered in the Directory tag. Both POSIX and Windows forms are allowed, unfortunately, but the backslash forms needs C-style escaping, and IIRC the Apache documentation recommends using forward slashes. Regards, Nicolas 2006/4/16, Graham Dumpleton [EMAIL PROTECTED]: I am sure I asked this a long time ago, but have forgotten all the details. On Win32 systems does req.filename set by Apache always use POSIX style forward slashes, ie., '/', to separate components of a directory? Thus:/some/path How does Apache indicate a drive letter when one is necessary? Is it:c:/some/path Does any of the above change based on whether forward or backward slashes are used in a Directory directive? Ie., Directory c:/some/path.../Directory? vs:Directory c:\\some\\path.../Directory Or does Apache not allow the latter anyway? If Apache does allow the latter, does that mean that req.hlist.directory is coming through set including backslashes rather than forward slashes. I want to get my head around this all again as at different times the values of req.filename and req.hlist.directory are used to determine the Python interpreter name. As highlighted in: http://issues.apache.org/jira/browse/MODPYTHON-161 If there is a mix of conventions, with user code also being able to affect these values, there may be no consistency and thus could end up with scenarios where a different interpreter to one than was expected will be used. Any help from Win32 users in understanding all this would be much appreciated. Thanks. Graham
Re: Pickling/unpickling top-level functions, classes etc.
OK, I'm +1 on the Won't fix status.I'm not proficient enough in the way unpickling works, but the fact that you cannot specify any namespace in which classes or functions names have to be resolved makes it quite clear that dynamic imports + unpickling functions and classes = not possible. Regards,Nicolas2006/3/29, Deron Meranda [EMAIL PROTECTED]: On 3/29/06, Graham Dumpleton [EMAIL PROTECTED] wrote: Are you okay with: http://issues.apache.org/jira/browse/MODPYTHON-81 Pickling/unpickling top-level functions defined in published module no longer works in mod_python 3.2 being resolved as Won't Fix and then closed?I agree that this seems to be something that is just not solvable without causing complete havoc with all thespecialized import and reload functionality, or resultingin a solution that is too fragile.It is just a limitation ofthe pickle mechanism.This of course doesn't mean that users wouldn't want to pickle these kinds of things.But that the burden in thosecases should be on them.It may be possible to solvethis for class instances (e.g., objects) by subclassing theUnpickler class and substituting a smarter find_class() method.But as for globals, functions, etc., it looks likeit may be much harder.The user may also be able to take advantage of theexternal object pickling (with persistent ids), but Ihaven't looked at them too closely. Regardless, there are lots of alternatives, so I haveno problem with mod_python not solving this one(although the mod_python documentation shouldclearly emphasize these pickling limitiations).-- Deron Meranda
Re: mod_python 3.3.0-dev-20060321 available for testing
2006/3/22, Graham Dumpleton [EMAIL PROTECTED]: Nicolas Lehuen wrote .. 2006/3/22, Nicolas Lehuen [EMAIL PROTECTED]: However I have a -1 on Python 2.2 with a LOT of test failures, but I guess we won't support Python 2.2 for mod_python 3.3 ? Sorry, my -1 was due to a configuration problem, everything works on Python 2.2. +1 for mod_python 3.3.0-dev-20060321 on Windows 2000 SP4 + ActivePython 2.2.3 + Apache 2.0.55 If you run the tests with the new importer, I would not have expected it to get very far with Python 2.2. This is because at one point it does: sys.meta_path.insert(0, _ModuleImporter()) Our understanding so far had been that sys.meta_path would only have appeared in Python 2.3, thus the import should have failed when that attribute was used. Graham Duh, I have forgotten to run the test using the new importer. All three results were therefore using the old importer. Regards, Nicolas
Re: mod_python 3.3.0-dev-20060321 available for testing
I've tested with and without the new importer on Windows XP SP2 +
Python 2.4.2 + Apache 2.2.0 and everything works except the
test_req_auth_type test, which signals a 500 error. This is what the
error_log contains about this test :
[Wed Mar 22 07:16:03 2006] [warn] mod_python
(pid=5140,interpreter='test_req_auth_type'): Module directory listed
in sys.path. This may cause problems. Please check code. Code file
being imported is C:\\projets\\mod_python\\test\\htdocs\\tests.py.
[Wed Mar 22 07:16:03 2006] [notice] mod_python
(pid=5140,interpreter='test_req_auth_type'): Importing module
'C:\\projets\\mod_python\\test\\htdocs\\tests.py'
[Wed Mar 22 07:16:03 2006] [crit] [client 127.0.0.1] configuration
error: couldn't check access. No groups file?: /tests.py
[Wed Mar 22 07:16:03 2006] [error] [client 127.0.0.1] No Authn
provider configured
The piece of code that emits the No groups file? seem to reside in
libhttpd.dll, a part of Apache 2.2, so I guess it's a problem with my
Apache setup. I'll try this on my Apache 2.0 setup on my PC at work
and let you know.
Regards,
Nicolas
2006/3/22, Jim Gallacher [EMAIL PROTECTED]:
mod_python-3.3.0-dev-20060321 is available for testing.
We are asking the mod_python development community for assistance in
testing the current development branch. Hopefully this will allow us to
catch new bugs or regressions early, and when we are ready for the next
release the beta cycle will be much shorter.
This snapshot addresses 33 issues since 3.2.7 was released, including
apache 2.2 support and the introduction of a new module importer.
The files are (temporarily) available here:
http://people.apache.org/~jgallacher/mod_python/dist/
Please download it, then do the usual
$ ./configure --with-apxs=/wherever/it/is
$ make
$ (su)
# make install
Then (as non-root user!)
$ make check
or if you prefer to run the tests the old way:
$ cd test
$ python test.py
Make a note of any failing tests.
If all the tests pass, give the new module importer a workout by
uncommenting line 328 in test/test.py:
#PythonOption('mod_python.future.importer *'),
and then re-run the tests.
$ make check
And see if any tests fail. If they pass, send a +1 to the list, if they
fail, send the details (the versions of OS, Python and Apache, the test
output, and suggestions, if any).
Thank you for your assistance,
Jim Gallacher
Re: New module importer. Was: Re: mod_python roadmap
If the new importer isn't on by default, I don't see any reason why you should not commit it to subversion, quite the contrary. Therefore I'm +1 on the subject. Regards, Nicolas 2006/3/19, Graham Dumpleton [EMAIL PROTECTED]: On 14/03/2006, at 12:23 PM, Jim Gallacher wrote: I find I work more effectively when I have deadlines to worry about (being a procrastinator by nature), so I thought I'd propose the following roadmap. Mar 20: 3.3-dev - snapshot for testing Apr 1: 3.2.9 - bugfix release May 1: 3.3-dev - snapshot for testing Jun 15: 3.3-dev - snapshot for testing Jul 15: 3.3 - feature freeze Aug 1: 3.3.0 - first 3.3 beta - branches/3.3.x created - work on trunk resumes - beta cycle proceeds independent of dev work Sep 15: 3.3.y - 3.3 final released (hopefully) For the development snapshots I'd just roll a tarball from trunk and make a call to the community for testing help. Hopefully we'll catch new bugs and regressions early so that the actual beta cycle will be much shorter. There would be *no* freeze during the snapshot tests. Work on trunk can continue while we wait for the test feedback. With the plan being to roll a tar ball on the 20th March, do people want me to incorporate the new module importer or not, such that it will be included in this snapshot and be available for testing? For background on the new importer see: https://issues.apache.org/jira/browse/MODPYTHON-143 and follow links given there to articles I have written or started writing and all the JIRA issues. The code for this is all ready, it just needs to be committed into the subversion repository. Note that just because the code would be part of the source code does not mean it will be used. Specifically, the code has been set up at the moment so the existing importer will still be used unless you explicitly configure mod_python to use the new importer. If you want to try the new module importer, you will be able to enable it for all Python interpreter instances created, or selected ones. Only after sufficient testing and tweaking as necessary, and after it has been deemed an acceptable solution would it be properly integrated into mod_python as the default. If people feel it isn't acceptable, it would be stripped out of code and someone else can have a go with coming up with a better alternative. Graham
Re: JIRA Housekeeping
Yup, I think it's the thing to do. Regards, Nicolas 2006/2/25, Jim Gallacher [EMAIL PROTECTED]: Now that JIRA is responding again I thought I'd update the status of some issues. I've created a new JIRA version for 3.2.8. Version 3.2 is still shown as unreleased. I assume the proper action is to rename it to 3.2.7 and mark it as released. Can someone confirm that this is the correct action? Nicolas? Grihsa? Jim
Re: mod_python 3.2.8 available for testing
Indeed :) 2006/2/22, Ron Reisor [EMAIL PROTECTED]: I know you're going ahead with 3.2.8 already, but I thought this would be interesting: +1 Mac OS X 10.4.5 Intel Core Duo, apache 2.0.55 mpm-prefork, python 2.4.2 cheers, Ron Ron Reisor [EMAIL PROTECTED] (RWR3) University of Delaware Information Technologies/Network and Systems Services Computing Center/192 South Chapel Street/Newark DE, 19716 pgp finger print: 0D 73 06 6F D3 6A 99 D3 F5 D5 6E FF 3B B9 7C 2C
Re: 3.2.8 summary / core group vote
OK, sorry, I was mislead by the fact that there were 3.2.5b binaries in the dist directory. Regards, Nicolas 2006/2/21, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: they're out there: http://www.apache.org/dist/httpd/modpython/win/3.2.8/ On Tue, 21 Feb 2006, Nicolas Lehuen wrote: Hi Grisha, Could you also put the Win32 binaries and make sure they are referenced on the download page ? We regularly have questions about where those binaries are, and I end up serving the files from my personal hosting solution, which is not really tailored for that. You can find the binaries here : http://nicolas.lehuen.com/download/mod_python TIA and regards, Nicolas 2006/2/21, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Resolved, the files have been placed on www.apache.org/dist, allowing time for mirror sync, then the web page update, then an announcement. Grisha On Mon, 20 Feb 2006, Graham Dumpleton wrote: +1 core vote Nicolas Lehuen wrote .. +1 core vote 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: +1 core vote Jim Gregory (Grisha) Trubetskoy wrote: Based on summary below, +1 from for putting it out there. Grisha Graham Dumpleton [EMAIL PROTECTED] +1 Mac OS X 10.4, apache 2.0.55 mpm-worker, python 2.3.5 (3.2.x SVN trunk) Nicolas Lehuen [EMAIL PROTECTED] +1 mod_python 3.2.8 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2 Barry Pederson [EMAIL PROTECTED] +1 FreeBSD 6.0, apache 2.0.55 mpm-prefork, python 2.4.2 Jim Gallacher [EMAIL PROTECTED] +1 Debian sid, apache 2.0.55 mpm-prefork, python 2.3.5
Re: How mod_python treats apache.OK/apache.DECLINED responsefromhandlers.
+1 Excellent summary, Graham. Maybe we could ask on the mod_pyhon mailing list who is stacking non-content handlers, especially if registered dynamically, and for what purpose ? This way we could make sure that no one actually relies on the current cludgy behaviour. But I agree with you, it's pretty sure that changing mod_python to align with the standard Apache behaviour should not meet popular disagreement, especially given the corner cases in which it really matters :). Regards, Nicolas 2006/2/22, Graham Dumpleton [EMAIL PROTECTED]: Grisha wrote .. If I understand this correctly, then +1. ...though I'm wondering if anyone will actually try to do something as arcane as dynamicaly registering non-content handers? :-) I agree, it might not be a totally realistic scenario, but then now that I have checked in a change to make req.handler writable, the system is becoming flexible enough that it may actually be reasonable to do it for some reason. Specifically, with the change to make req.handler writable, instead of using SetHandler/AddHandler to have mod_mime internally set req.handler to mod_python, you could define your own type handler which did it. def typehandler(req): if os.path.splitext(req.filename)[1] == .py: req.handler = mod_python req.add_handler(PythonHandler,mod_python.publisher) return apache.OK return apache.DECLINED You might even at the same time want to register a fixup handler to do stuff prior to the response phase being run: def typehandler(req): if os.path.splitext(req.filename)[1] == .py: req.handler = mod_python req.add_handler(PythonFixupHandler,manage_session_object) req.add_handler(PythonHandler,mod_python.publisher) return apache.OK return apache.DECLINED For example, you might introduce a fixup handler which ensures that a session object is created and put in req.session. This is a lot cleaner than what most people do, which is to put a call to the session manager code in every single published function. Graham On Tue, 21 Feb 2006, Jim Gallacher wrote: Nice summary. +1 for the change. Jim Graham Dumpleton wrote: Jim Gallacher wrote .. I don't have alot more to say on these last 2 emails. I think you are on the right path here. Okay, I think I have a good plan now. To summarise the whole issue, the way Apache treats multiple handlers in a single phase for non content handler phases is as follows: PostReadRequestHandler RUN_ALL TransHandler RUN_FIRST MapToStorageHandler RUN_FIRST InitHandler RUN_ALL HeaderParserHandler RUN_ALL AccessHandlerRUN_ALL AuthenHandlerRUN_FIRST AuthzHandler RUN_FIRST TypeHandler RUN_FIRST FixupHandler RUN_ALL LogHandler RUN_ALL RUN_ALL means run all handlers until one returns something other than OK or DECLINED. Thus, handler needs to return DONE or an error to have it stop processing for that phase. RUN_FIRST means run all handlers while they return DECLINED. Thus, needs handler to return OK, DONE or error to have it stop processing for that phase. Where multiple handlers are registered within mod_python for a single phase it doesn't behave like either of these. In mod_python it will keep running the handlers only while OK is returned. Returning DECLINED causes it to stop. This existing behaviour can be described (like mod_perl) as stacked handlers. Having non content handler phases behave differently to how Apache does it causes problems. For example things like a string of authentication handlers which only say OK when they handle the authentication type, can't be implemented properly. In Apache, it should stop the first time one returns OK, but in mod_python it will keep running the handlers in that phase. In summary, it needs to behave more like Apache for the non content handler phases. In respect of the content handler phase itself, in practice only one handler module is supposed to implement it. At the Apache level there is no concept of different Apache modules having goes at the content handler phase and returning DECLINED if they don't want to handle it. This is reflected in how in the type handler phase, selection of the module to deliver content is usually done by setting the single valued req.handler string. Although, when using mod_python this is done implicitly by setting the SetHandler/AddHandler directives and mod_negotiation then in turn setting req.handler to be mod_python for you. Because mod_python when executed for the content handler phase is the only thing generating the content, the existing mechanism of stacked handlers and how the status is handled is fine
Re: mod_python 3.2.8 available for testing
Oops, I've sent this mail a bit too fast... As usual, all three binary versions are available here : http://nicolas.lehuen.com/download/mod_python/ Regards, Nicolas 2006/2/20, Nicolas Lehuen [EMAIL PROTECTED]: +1 mod_python 3.2.8 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2
Re: 3.2.8 summary / core group vote
+1 core vote 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: +1 core vote Jim Gregory (Grisha) Trubetskoy wrote: Based on summary below, +1 from for putting it out there. Grisha Graham Dumpleton [EMAIL PROTECTED] +1 Mac OS X 10.4, apache 2.0.55 mpm-worker, python 2.3.5 (3.2.x SVN trunk) Nicolas Lehuen [EMAIL PROTECTED] +1 mod_python 3.2.8 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2 Barry Pederson [EMAIL PROTECTED] +1 FreeBSD 6.0, apache 2.0.55 mpm-prefork, python 2.4.2 Jim Gallacher [EMAIL PROTECTED] +1 Debian sid, apache 2.0.55 mpm-prefork, python 2.3.5
Re: 3.2.8 summary / core group vote
Hi Michel, I've tested your patch on Win32 + Apache 2.2 and it mostly works (compilation + unit tests) - except for the changes in the PSP lexer parser. They now includ unistd.h which was previously hidden for Win32 platforms.It looks like you have generated those files from the .l grammar using flex, maybe it was an old version ? In any case, I don't understand what those changes have to do with the problem you reported. So my position would be to integrate your patch except for the PSP part. Regards, Nicolas 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: Hi Michel, Please create a JIRA issue. We are much less likely to loose track of it if it's in the issue tracker. Thanks, Jim Michel Jouvin wrote: 0 Tru64 5.1B-3, Apache 2.0.55, worker MPM, Python 2.4.1 I have a problem to build mod_pyton on Tru64 with the Python config I have. This is basically due to the fact that Python.h should be included first as its defines some macros used by standard includes. When included at the end of the includes, this results to some conflicts because the same include is included twice with a different macro value. I reported this a couple of months ago during 3.2 beta. Do you want me to resubmit this problem via JIRA ? Or just to resend the required fixes ? Cheers, Michel --On lundi 20 février 2006 16:18 -0500 Graham Dumpleton [EMAIL PROTECTED] wrote: +1 core vote Nicolas Lehuen wrote .. +1 core vote 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: +1 core vote Jim Gregory (Grisha) Trubetskoy wrote: Based on summary below, +1 from for putting it out there. Grisha Graham Dumpleton [EMAIL PROTECTED] +1 Mac OS X 10.4, apache 2.0.55 mpm-worker, python 2.3.5 (3.2.x SVN trunk) Nicolas Lehuen [EMAIL PROTECTED] +1 mod_python 3.2.8 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2 Barry Pederson [EMAIL PROTECTED] +1 FreeBSD 6.0, apache 2.0.55 mpm-prefork, python 2.4.2 Jim Gallacher [EMAIL PROTECTED] +1 Debian sid, apache 2.0.55 mpm-prefork, python 2.3.5 * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: 3.2.8 summary / core group vote
Er... I should say that the changes in _pspmodule.c are fine, so they should be integrated. The problems are in the changes brought to the files that are generated by flex. First, they break on Win32, second, since the .l source file is unchanged, next time flex is run the changes will be overwritten. Regards, Nicolas 2006/2/21, Nicolas Lehuen [EMAIL PROTECTED]: Hi Michel, I've tested your patch on Win32 + Apache 2.2 and it mostly works (compilation + unit tests) - except for the changes in the PSP lexer parser. They now includ unistd.h which was previously hidden for Win32 platforms.It looks like you have generated those files from the .l grammar using flex, maybe it was an old version ? In any case, I don't understand what those changes have to do with the problem you reported. So my position would be to integrate your patch except for the PSP part. Regards, Nicolas 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: Hi Michel, Please create a JIRA issue. We are much less likely to loose track of it if it's in the issue tracker. Thanks, Jim Michel Jouvin wrote: 0 Tru64 5.1B-3, Apache 2.0.55, worker MPM, Python 2.4.1 I have a problem to build mod_pyton on Tru64 with the Python config I have. This is basically due to the fact that Python.h should be included first as its defines some macros used by standard includes. When included at the end of the includes, this results to some conflicts because the same include is included twice with a different macro value. I reported this a couple of months ago during 3.2 beta. Do you want me to resubmit this problem via JIRA ? Or just to resend the required fixes ? Cheers, Michel --On lundi 20 février 2006 16:18 -0500 Graham Dumpleton [EMAIL PROTECTED] wrote: +1 core vote Nicolas Lehuen wrote .. +1 core vote 2006/2/20, Jim Gallacher [EMAIL PROTECTED]: +1 core vote Jim Gregory (Grisha) Trubetskoy wrote: Based on summary below, +1 from for putting it out there. Grisha Graham Dumpleton [EMAIL PROTECTED] +1 Mac OS X 10.4, apache 2.0.55 mpm-worker, python 2.3.5 (3.2.x SVN trunk) Nicolas Lehuen [EMAIL PROTECTED] +1 mod_python 3.2.8 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 mod_python 3.2.8 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2 Barry Pederson [EMAIL PROTECTED] +1 FreeBSD 6.0, apache 2.0.55 mpm-prefork, python 2.4.2 Jim Gallacher [EMAIL PROTECTED] +1 Debian sid, apache 2.0.55 mpm-prefork, python 2.3.5 * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: Getting Started on mod_python 3.3.
Based on today's traffic on the mailing lists, I think that we should go for a short-term 3.2.8 release of mod_python, with certified Apache 2.2 support on multiple platforms. The code is only there but I suppose we'll need a lot of testing, so maybe we could expect to release this in a month or two (given that the Win32 source code is not even available right now). Regards, Nicolas 2006/2/14, Nicolas Lehuen [EMAIL PROTECTED]: 2006/2/14, Graham Dumpleton [EMAIL PROTECTED]: [...] If we want to go down the path of having interim 3.2 bug rollup releases while 3.3 is being developed, might I suggest that we target the following for such a release in the near future. MODPYTHON-77 The Simplified GIL Aquisition patches. MODPYTHON-78 Apache 2.2 patches. MODPYTHON-94 Support for optional mod_ssl functions on request object. MODPYTHON-113 Make PythonImport use apache.import_module() via CallBack method. MODPYTHON-119 DBM Session test patches. MODPYTHON-122 Bash 3.1.X configure patches. I know that MODPYTHON-94 isn't a bug fix, but a few people have been after this one. Also MODPYTHON-113 may not seem important, but will mean that any test package I make available for new importer will work properly in all cases where module imports occur. Anyway, after trolling through JIRA, these seemed to be the important ones to me, but other might have other suggestions. Now, the question is how we manage this. Do we concentrate on these only in the trunk and get them out of the way first as a 3.2.X release, holding back any changes to test framework? Or do we merge such changes from trunk on a case by case basis in 3.2.X branch? Graham I was thinking about working on the new test framework in parallel of real work, away from the trunk (in my /branches/nlehuen directory). I don't think it will be too hard to track down the changes in the trunk tests and bring them back in the new test framework, but I may be wrong. One the new tests are available, I'll merge them back in the trunk. So I guess it's not necessary to hold back the next release do to the tests, and it may be a good exercise to due a few 3.2.x releases in a short period of time before doing the 3.3.x release. Regards, Nicolas
Re: Getting Started on mod_python 3.3.
2006/2/14, Nicolas Lehuen [EMAIL PROTECTED]: Based on today's traffic on the mailing lists, I think that we should go for a short-term 3.2.8 release of mod_python, with certified Apache 2.2 support on multiple platforms. The code is only there but I suppose we'll need a lot of testing, so maybe we could expect to release this in a month or two (given that the Win32 source code is not even available right now). Regards, Nicolas Doh ! I've found the source code for Win32. I'll try to build it ASAP and give mod_python a try. Regards, Nicolas
Testing mod_python SVN trunk with Apache 2.2 on Win32
Hi, I've built Apache 2.2 and tested mod_python SVN trunk with it. The two register_cleanup tests fail. Apparently it's because the test code registers a cleanup function giving the current request as parameter. Of course when the cleanup function is called, the request object is no longer valid, and Apache segfaults. Fixing this is only a matter of fixing the test code, yet I wonder how this code could properly run on Apache 2.0.55. Maybe the request object was not properly destroyed and this has been fixed in Apache 2.2 ? This also shows that we should document the fact that the current request object should not be passed directly or indirectly to the *.register_cleanup functions. Maybe we could implement a little test in those function to make sure it is not passed directly ? Those two failures aside, the rest of the tests are OK. Regards, Nicolas
Cool feature from mod_perl : Configure Apache with Perl
Hi, I'm currently reading the feature section from mod_perl. Initially, I was trying to find information about how they cope with multithreading, multiple interpreter instantiation and code reloading, but I stumbled upon this : http://perl.apache.org/start/tips/config.html Now, I can't stand Perl, but this feature is quite cool, isn't it ? Would it be difficult to implement, say, in mod_python 4.0 ? Regards, Nicolas
For the curious : how mod_perl handles threading
http://perl.apache.org/docs/2.0/user/intro/overview.html#Threads_Support Regards, Nicolas
Re: For the curious : how mod_perl handles threading
It seems that bu dfdefault Perl is not thread safe, and that they have to jump through all those hoops to ensure thread safety. There is no real lesson for mod_python, I just wanted to know how they solved this rather difficult problem. Not instantiating one interpreter per name per thread and using an interpreter pool may be an interesting optimisation, though. But I guess it's a rather complicated one. Regards, Nicolas 2006/2/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: On Mon, 13 Feb 2006, Nicolas Lehuen wrote: http://perl.apache.org/docs/2.0/user/intro/overview.html#Threads_Support Which part of it - the pool of interpreters? Are they doing it out of necessity, i.e. there is no way to run multiple threads in Perl like we do in Python because of Python's GIL? Grisha
Refactoring of the test suite
Hi, There is something I'd like to do for the 3.3 version : it is to refactor the test suite. It's more a chore than real development, but the current test suite is slowly becoming big and quite difficult to maintain. What I'd like to do is simply split the test runner and the published tests in various parts, matching the different functionalities that are tested. What would be great, then, is to introduce platform-specific tests, of even MPM-specific tests, so that we could exercise some parts of the code that are only available with a threaded MPM (for example, the MemorySession implementation, or the publisher reload test which I erroneously implemented with a threaded MPM in mind). One thing that I'd like to settle with you is whether you are OK to split the big PerRequestTestCase class into multiple classes, to ease maintenance and configuration switches. This has the downside that Apache server is likely to be stopped and restarted a few more times during the tests, so the tests will take more time to run. On the other hand, I'd like to introduce a way to select the tests to run when launching the test suite, so that we won't have to wait for the whole test suite to pass while debugging a given problem. Ah, and one thing I'd definitely get rid of - usage of backquotes like in `rsp`. I really don't like this magic quoting thingy, it reminds me all too much of PHP :). Like Graham says : comments ? Regards, Nicolas
Re: mod_python 3.2.7 available for testing
Hi David, One thing I don't understand is why you have foreign PythonOption definitions in your test setup. Are you sure you have re-created a clean testconf.py file from testconf.py.in ? I've write a little bit of documentation in test/README, please try to follow it. It seems to me that the test failures all come from this strange test configuration. It looks like mod_python is interfering with some already existing filters, like MultiView. Note that there is a bug under Win32 which I should have reported way before... In some situation, when an exception is raised in mod_python during normal operation, Apache will segfault the next time is is stopped or restarted. It only occurs with a specific exception is raised, maybe when a handler module fails to load. The reason why I haven't reported it yet is that it only occured on my development computer, not my production servers, so I thought it was something caused by my setup. Apparently you have the same problem, hence the segfault when the Apache server is stopped. I'll have to investigate this problem more thoroughly. So please try to re-run the test in a clean slate environment. It'd be interesting to see why and how your current configuration contaminates the test configuration, though. Regards, Nicolas 2006/2/8, David Fraser [EMAIL PROTECTED]: Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: I think we have enough +1's. (If someone could tally them up in a single e-mail, that'd be great.) Should we start a core-group vote, or wait some more? On the bash issue - I think we can leave it as is, the affected distros will just have to maintain a patch in their build systems. Let's vote. Here is the test summary: +1 Debian (sid), Apache 2.0.55-prefork, Python 2.3.5 +1 Debian (sarge), Apache 2.0.54-worker, Python 2.3.5 +1 Debian (sarge), Apache 2.0.54-prefork, Python 2.3.5 +1 Debian (testing, aka, etch), Apache 2.0.55-worker, Python 2.3.5 +1 Fedora Core 4, Linux 2.6.15, Apache 2.0.54, Python 2.4.1 +1 FreeBSD 4.9 , Apache 2.0.50 (prefork), Python 2.3.4 +1 FreeBSD 4.9 , Apache 2.0.55 (prefork), Python 2.4.2 +1 MacOSX 10.4.4 PPC, Apache-2.0.55-prefork, Python-2.4.2 +1 Slackware 10.1, Apache 2.0.55 (mpm-prefork), Python 2.4 +1 Solaris 10 Sparc, Apache-2.0.55-prefork, Python-2.4.2 +1 Ubuntu 5.10 Breezy (amd64), Apache 2.0.54-worker, Python 2.4.2 +1 Windows 2000 SP4, Apache/2.0.55 + Python/2.2.3 +1 Windows 2000 SP4, Apache/2.0.55 + ActivePython/2.3.5 +1 Windows XP SP2, Apache/2.0.55 + ActivePython/2.4.2 With configure fixed manually to deal with bash bug: +1 Gentoo 2005.1 (amd64), Apache 2.0.55-prefork, Python 2.4.2 Plus we have the following tests from the svn revision which corresponds to 3.2.7: +1 trunk rev 374709 FreeBSD 6.0 Apache 2.0.55-prefork, Python 2.4.2 I know its already been decided, but I thought I'd add another test for Windows (seeing as Nicolas is the only one who has tested so far...) This was with the installers Nicolas built, running the tests out of svn version 375881 +1 Windows XP SP2, Apache/2.0.54, Python 2.4.1 Unfortunately I got a failure too: -1 Windows 2000 SP4, Apache/2.0.49, Python 2.3.3 This even incorporated Apache segfaults ... when trying to stop with server.register_cleanup and apache.register_cleanup Other failures: req.add_handler() when handler list is empty PythonOption override, remove and remove2 PythonOutputFilter test_internal In all, I got 6 failures on the first 45 tests and 3 on the last 6. The PythonOption test failures look like they're conflicting with some PythonOptions in my default Apache config (since the assertion errors include things like jToolkit and jLogbook), so this seems to be an error in the test setup that its not independent of that. The PythonOutputFilter test also seems to be a test error, as it is returning the *contents* of test.py in uppercase instead of TEST OK in uppercase... Similarly test_accesshandler_add_handler_to_empty_hl is returning the contents of test.py I wonder if the segfaults on cleanup relate to my Apache version, if so I can try upgrading. Am I doing something wrong in the testing? Logs attached (zipped as the inclusion of test.py a few times make them quite big)... David
Re: mod_python 3.2.7 available for testing
OK so my core group vote is +1 for this release. It has been tested on a wide array of OSes, both threaded and forked MPMs, Python 2.2, 2.3 and 2.4, so I guess it's okay. A threaded test on MacOSX and Solaris would have been great but maybe the recommended MPM on those platform is the forked one, so we don't have to worry about those. Regards, Nicolas 2006/2/7, Jim Gallacher [EMAIL PROTECTED]: Gregory (Grisha) Trubetskoy wrote: I think we have enough +1's. (If someone could tally them up in a single e-mail, that'd be great.) Should we start a core-group vote, or wait some more? On the bash issue - I think we can leave it as is, the affected distros will just have to maintain a patch in their build systems. Let's vote. Here is the test summary: +1 Debian (sid), Apache 2.0.55-prefork, Python 2.3.5 +1 Debian (sarge), Apache 2.0.54-worker, Python 2.3.5 +1 Debian (sarge), Apache 2.0.54-prefork, Python 2.3.5 +1 Debian (testing, aka, etch), Apache 2.0.55-worker, Python 2.3.5 +1 Fedora Core 4, Linux 2.6.15, Apache 2.0.54, Python 2.4.1 +1 FreeBSD 4.9 , Apache 2.0.50 (prefork), Python 2.3.4 +1 FreeBSD 4.9 , Apache 2.0.55 (prefork), Python 2.4.2 +1 MacOSX 10.4.4 PPC, Apache-2.0.55-prefork, Python-2.4.2 +1 Slackware 10.1, Apache 2.0.55 (mpm-prefork), Python 2.4 +1 Solaris 10 Sparc, Apache-2.0.55-prefork, Python-2.4.2 +1 Ubuntu 5.10 Breezy (amd64), Apache 2.0.54-worker, Python 2.4.2 +1 Windows 2000 SP4, Apache/2.0.55 + Python/2.2.3 +1 Windows 2000 SP4, Apache/2.0.55 + ActivePython/2.3.5 +1 Windows XP SP2, Apache/2.0.55 + ActivePython/2.4.2 With configure fixed manually to deal with bash bug: +1 Gentoo 2005.1 (amd64), Apache 2.0.55-prefork, Python 2.4.2 Plus we have the following tests from the svn revision which corresponds to 3.2.7: +1 trunk rev 374709 FreeBSD 6.0 Apache 2.0.55-prefork, Python 2.4.2 Jim
Re: Change to test_Session_Session_conf() of test/test.py.
2006/2/4, Graham Dumpleton [EMAIL PROTECTED]:
Jim, Nicolas
Would it make sense to change test_Session_Session_conf() function in
unit tests to something like:
def test_Session_Session_conf(self):
import tempfile
tempdir = tempfile.gettempdir()
database = os.path.join(tempdir,mp_sess_test.dbm)
c = VirtualHost(*,
ServerName(test_Session_Session),
DocumentRoot(DOCUMENT_ROOT),
Directory(DOCUMENT_ROOT,
PythonOption('session_dbm %s' %
database),
SetHandler(mod_python),
PythonHandler(tests::Session_Session),
PythonDebug(On)))
return str(c)
Ie., explicitly set session_dbm to some other location than default.
Without this the test can fail if main Apache is run as different
account to what test is run as. I put database in /tmp under a
different
name, but might be better somewhere in the test directory of the source
code.
Does this make sense?
Of course, no problem ! I agree with you that it would be better if
the session file was in the local directory, making sure it is removed
before the test runs. Maybe the unit test could open the session dbm
after the test has run to check whether the session is saved, for
additional safety.
Regards,
Nicolas
That gets rid of one of the failures, now for the others. :-)
Graham
Re: 3.2.6 or not
+1 trunk rev 374588 + Apache/2.0.55 + Python/2.2.3 + Windows 2000 SP4 +1 trunk rev 374588 + Apache/2.0.55 + ActivePython/2.3.5 + Windows 2000 SP4 +1 trunk rev 374588 + Apache/2.0.55 + ActivePython/2.4.2 + Windows XP SP2 All three installers for win32 are available at http://nicolas.lehuen.com/download/mod_python Regards, Nicolas 2006/2/3, Daniel J. Popowich [EMAIL PROTECTED]: Jim Gallacher writes: Graham Dumpleton wrote: To confirm Jim's arithmetic anyway, I say -1 on 3.2.6 as it stands. As to 3.2.7, I say +1, subject to removal of problematic test case as already raised and with us at least confirming tests run OK for version out of SVN prior to packaging. Oh, so *that's* where you said we should disable the test. ;) I've disabled the publisher_cache test and the connection handler fix has been checked in. We are good to go for 3.2.7. As Graham suggests perhaps we could get some people to confirm the current version in SVN trunk prior to packaging. A couple of thumbs up from some FreeBSD'ers would be especially nice since the connection handler problem seemed to be most prevalent on that platform. If everything looks ok I'll create the 3.2.7 tarball sometime Friday. Just checked out the trunk... +1 trunk + Apache/2.0.55 + Python/2.3.5 + Debian/testing(etch)
Re: Python 2.2 support
I am hereby happy to tell you that by removing the call to enumerate() in the publisher code, the whole test suite passes on Python 2.2 without any further patch or hack. I've checked in the modification which this time should not pose any problem since it's pretty basic and non intrusive. Regards, Nicolas 2006/2/2, Graham Dumpleton [EMAIL PROTECTED]: On 02/02/2006, at 5:42 PM, Nicolas Lehuen wrote: That's it ! People with Python 2.2 could use PythonImport mod_python.python22 INTERPRETER_NAME in their configuration file to make sure mod_python supports Python 2.2. The only problem is the need to provide an interpreter name, which complicates things a little bit in the case of the test suite. Then again, the only thing which prevents Python 2.2 support right now is the use of enumerate(), so we could just check whether we could do without enumerate() and support Python 2.2 out of the box. The code didn't used to use enumerate() in that one place and it worked then. Just means having to have a counter that is manually incremented. Don't have access to code right now, else would post changed code. :-) Graham
Re: 3.2.6 or not
My official vote is eventually -1 for 3.2.6, see the previous discussion for why I've changed my mind. However I'm +1 on releasing 3.2.7 without a restrained testing period, not a long one like for 3.2.6. Regards, Nicolas 2006/2/2, Jim Gallacher [EMAIL PROTECTED]: I know you said no discussion Grisha, but can I have 2 ballots? ;) -1 If Graham thinks his conn handler fix is good, let's do 3.2.7 today. +1 If Graham is not sure, we release 3.2.6 now as is, and do a 3.2.7 bugfix in the next 4 to 6 weeks after digging into _conn_read issue further. So, I guess that makes my official vote a +0. Over to you Graham. No pressure though. :) Jim (Dang, it makes me feel dirty to waffle on my first offical vote that way). Gregory (Grisha) Trubetskoy wrote: OK, I know we've had some votes on this before, but I'd like to put this in a separate thread where it's not intermixed with all kinds of other things. This is a vote for the core group. We can release the 3.2.6 tarball as is or fix the connection handler bugs (there are two of them - the buffer pointer and eagain condition Graham tracked down) and release a 3.2.7 (or 3.2.6.1). The rationale for disregarding those known issues is that the connection handler is hardly used by anyone. The rationale for NOT disregarding is that we claim this to be a stable release, and given our slow release cycle, I imagine 3.2.6 will be around for a while. Anyhow - *the core group* (you know who you are), if you think 3.2.6 should be released as is, send in your +1. Let's keep this thread strictly a vote, without it turning into a discussion (we can discuss things in other threads). My official vote is +0. (To see what this means read http://httpd.apache.org/dev/guidelines.html) Grisha
Re: Worrying code in mod_python.publisher module importer.
2006/2/2, Graham Dumpleton [EMAIL PROTECTED]: Okay, false alarm (I think). Have got myself worked up over nothing. I missed something very important: timestamp = fstat(opened.fileno())[-2] That is the '[-2]' in the above. I feel like a goose now. I still though question why file/fstat is done and not stat/file though. Ie., why open the file to stat it? Graham Well, I thought that if the file was modified, we needed to open it anyway, but you're right, that's optimising for a minority case. We might as well use stat and open the file only if it has changed. I've wrote an alternative publisher a few months ago that overloaded this behaviour in the module cache to use req.finfo[apache.FINFO_MTIME] as the file modification time, thus saving us a call to fstat or stat entirely. I've stopped using this publisher because I thought that using the standard publisher was a better way to see how we could improve it, but anyway, I could back port this trick. If I don't get burned down by the flak I'm currently getting on the Python 2.2 issue, that is ;). Regards, Nicolas
Re: Worrying code in mod_python.publisher module importer.
2006/2/2, Graham Dumpleton [EMAIL PROTECTED]: Note that up until now I hadn't even looked over how this new module importer was implemented. I knew it wasn't going to solve various of the existing module importer problems and I knew it was actually going to introduce some new issues that would have to be worked around, but now that I have started to document these new issues for inclusion in my module importer issues list and when I see other possible problems like the above, I am really starting to wander if it is really a good idea letting this interim solution to module importing problems be released. Comments? Graham You know, Graham, I'm very frustrated about this because we decided not to go any further on the module importer issue until we reach 3.3. Hence, I have stopped any development on this level and kept the code as is (i.e. in a working state), hoping that the 3.2 release would come soon and that we would be able to move on quickly. More than six months later we're still at the same point and now you're beginning to ask questions about the interim solution. Well, indeed, it's an interim solution, but it works and fixes a lot of bugs. It's not perfect, we'll surely have some people asking us how to import one published module from another one (I had wrote some code to support that, but it was refused), but it was never supposed to last long. Anyway, I'd like to point out that I've been using this publisher in various professional projects for months now without having any problems. It's not like we are releasing something flaky. The only problem is that apache.import_module is still as crappy as ever and that we don't have any grand unified theory of module importing that would support both handlers and published modules. Regards, Nicolas
Re: Python 2.2 support
Quick response, because it's 4:36 AM here, I just woke up to feed my daughter and took all this flak, but I need to sleep :). I guess that's the problem of having a round-the-planet development team, between those in America, Europe, Asia and Australia (nobody from Antarctica yet ?) Graham : Considering that Grisha wanted your fix for the connection handler included in the 3.2 release, I thought that the 3.2.6 release was DOA, so I didn't feel bad about checkin this in. Anyway, Jim has the good idea there : We've got a clean 3.2.6 tag, so we can release it and either branch on it or stay on the trunk. We don't have to stop everything just because we want to release something, Subversion is here to help. Daniel : I think that from python22 import * is no more troublesome as the from __future__ import generators that you have to put in any source code that feature generators and needs to run from Python 2.2 and upward. But that's a matter of taste. Maybe we could do something like you wrote, for example in a mod_python.python22support module that Python 2.2 users would import using PythonImport ? The only problem is that the current solution passes all the test suite from Python 2.2 to Python 2.4 (I've yet to test it on Python 2.5), whereas a solution using PythonImport or any custom tweaking of the configuration would require a special case in the test suite. Now I'm going back to sleep. Regards, Nicolas 2006/2/2, Daniel J. Popowich [EMAIL PROTECTED]: Nicolas Lehuen writes: I've just checked in some changes to the Python source code in order to support Python 2.2. Now the test suite runs successfully on Python 2.2.3 on Windows 2000. I've checked that no regressions were introduced in later Python versions, too. The changes are pretty simple : each Python module now features a from python22 import *. The mod_python.python22 module just reimplements new builtins from Python 2.3. It turns out that the only missing builtin for now is enumerate(). The tests module, containing a few tests for generators, has to sport a from __future__ import generators line. I also had to change mod_python.cache which used time.strptime so that it uses rfc822.parsedate, now. I've did this because the guys from Nokia use Python 2.2 and mod_python 3.1.3. They spotted memory leaks which are likely to be fixed in mod_python 3.2.X, but they could not upgrade if Python 2.2 was not supported. But does this scale over the life of a project? Every new python module has to include 'from python22 import *'?? While never shying away from a decent hack myself, this is a hack that affects everyone, not just those that need the hack, which, for me, screams volumes. Certainly, which version of python mod_python is based on should not be taken lightly. Was it formally decided to make mod_python dependent on 2.3+? If not, then perhaps uses of enumerate, etc. should not be used. If a formal decision was made, then it's a done deal, right? If not and uses of 2.3 have slipped in then perhaps it's a done deal anyway because no one can stomach the thought of taking out the 2.3-isms at this late date. Regardless, I do not think it is within the scope of mod_python developers to keep users forward-compatible with the underlying python version. Sorry, but IMHO, this is not scalable software engineering. That said, I don't have a problem helping 2.2 users write their own hacks. For example, let's take enumerate...another way of solving this problem without touching any mod_python code would be a suggestion in the FAQ to write a module, say, python22hacks.py: # python22hacks.py # # This is unsupported software offered to mod_python users who are # stuck using python 2.2. # # Install by placing this module in the same directory with other # mod_python modules and add the following to your apache config: # #PythonImport python22hacks INTERPRETER_NAME # # More disclaimers, blah, blah... import __builtin__ as hack hack.enumerate = lambda s: zip(xrange(len(s)), s) This way we don't have to touch every module of source code and the onus is on the few who need it rather than the many who don't or those who have to maintain it. Cheers, Daniel Popowich --- http://home.comcast.net/~d.popowich/mpservlets/
Re: Python 2.2 support
2006/2/2, Jim Gallacher [EMAIL PROTECTED]: If a formal decision was made, then it's a done deal, right? If not and uses of 2.3 have slipped in then perhaps it's a done deal anyway because no one can stomach the thought of taking out the 2.3-isms at this late date. My impression is that there was never really a discussion on this issue. Some 2.3-isms got used, so it was decided that 2.2 was not supported. There likely should have been a more formal discussion on whether this is right time to drop 2.2 support. Certainly we haven't done any testing using python 2.2 so even with the hack I don't think we can comfortably claim that that version is supported. Note that I have ran successfully the unit tests on Python 2.2, 2.3 and 2.4 before checking this hack in. Granted, this is not a guarantee that Python 2.2 is supported, but given our tests coverage, this is pretty good. Regards, Nicolas
Re: Worrying code in mod_python.publisher module importer.
OK, I've changed cache.py so that it uses stat() then open() the file if it needs to be reloaded. I've also added a unit test that makes sure the module cache is behaving as expected. Graham, I don't think the stat() / open() / fstat() sequence is required. How would that improve accuracy ? Regards, Nicolas
Re: Python 2.2 support
OK, I've reverted my changes. I left python22.py in place, because I still hope to be able to use it with PythonImport. The only problem is being able to define it in the unit tests. Regards, Nicolas 2006/2/2, Nicolas Lehuen [EMAIL PROTECTED]: 2006/2/2, Jim Gallacher [EMAIL PROTECTED]: If a formal decision was made, then it's a done deal, right? If not and uses of 2.3 have slipped in then perhaps it's a done deal anyway because no one can stomach the thought of taking out the 2.3-isms at this late date. My impression is that there was never really a discussion on this issue. Some 2.3-isms got used, so it was decided that 2.2 was not supported. There likely should have been a more formal discussion on whether this is right time to drop 2.2 support. Certainly we haven't done any testing using python 2.2 so even with the hack I don't think we can comfortably claim that that version is supported. Note that I have ran successfully the unit tests on Python 2.2, 2.3 and 2.4 before checking this hack in. Granted, this is not a guarantee that Python 2.2 is supported, but given our tests coverage, this is pretty good. Regards, Nicolas
Re: Python 2.2 support
2006/2/2, Graham Dumpleton [EMAIL PROTECTED]: Nicolas Lehuen wrote .. OK, I've reverted my changes. I left python22.py in place, because I still hope to be able to use it with PythonImport. The only problem is being able to define it in the unit tests. I plead dumb. What is the connection to PythonImport? My only guess at the moment is that it does something like I do in my new importer, which is to use PythonImport to import a module which goes in and fiddles with the contents of mod_python.apache/publisher to patch in my new code before any request handlers get a chance to be called. In this way I don't have to be patching the actual mod_python source code. Graham That's it ! People with Python 2.2 could use PythonImport mod_python.python22 INTERPRETER_NAME in their configuration file to make sure mod_python supports Python 2.2. The only problem is the need to provide an interpreter name, which complicates things a little bit in the case of the test suite. Then again, the only thing which prevents Python 2.2 support right now is the use of enumerate(), so we could just check whether we could do without enumerate() and support Python 2.2 out of the box. Regards, Nicolas
Re: 3.2.6 test period - how long do we wait?
OK, so shall we schedule the 3.2.x release for 2007, then ? As for the Apache 2.2 version, what if we roll in your suggested patch, Jim, then discover a bunch of problem related to it during the beta tests ? Will we wait until they are all fixed to release the 3.2 version ? Apache 2.2 is quite new so we'll likely to have to squish bugs, due for example to new interaction between Apache filters and mod_python. That's a wild guess but filters have been modified in Apache 2.2 so I'm sure something evil lurks there. bitterOr we could simply forget about making the release one day and tell every user to use the latest snapshot from subversion. Sorry to be like that, but we have users out there that would be perfectly happy with the current state of the 3.2.6 version, and a lot of our answers on the mailing list are yup, we know this bug, it's already been fixed one year ago, but don't worry, you'll get the bugfix soon enough./bitter Once again, it seems that no regression have been introduced in 3.2.6 vs 3.1.4, so we should release it ASAP and try to keep a steady release rythm afterwards. When we'll get momentum we'll solve a bunch of problem pretty fast, but it's been a year now that we are paralysed by perfectionism. What could be worse than leaving our users out there with the current 3.1.4 version ? Regards, Nicolas 2006/1/31, Jim Gallacher [EMAIL PROTECTED]: I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? If that is the case I wonder if we should roll in the changes to support apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The original macro is: #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ || (s) == APR_OS_START_SYSERR + NO_ERROR) The discussion on httpd-dev suggested that this macro should be substituted with a simple test such as if (rc != APR_SUCCESS), and the '||' condition was not likely used. So that we are making the fewest possible changes to our current 3.2 codebase, I'd suggest reimplenting APR_STATUS_IS_SUCCESS in our code, and then removing it 3.3. This will give us lot's of time as we work on 3.3 to discover if there are any problems droping the APR_OS_START_SYSERR part of the test. Jim
Re: [jira] Created: (MODPYTHON-114) Problems with PythonPath directive.
2006/1/26, Mike Looijmans [EMAIL PROTECTED]: Two comments: 1. (bug): The acquire() call should be *outside* the try ... finally block. You do not want to release a lock that you did not aquire. 2. (optimization): If you are not planning to change the path, you do not have to aquire the lock. Aquiring a lock is quite expensive, so the double-check will have much less impact on performance. if config.has_key(PythonPath): pathstring = config[PythonPath] if not _path_cache.has_key(pathstring): # acquire must be done outside the try block _path_cache_lock.acquire() try: # double-check, because two threads might come # to the same conclusion up until here. if not _path_cache.has_key(pathstring): newpath = eval(pathstring) _path_cache[pathstring] = None sys.path[:] = newpath finally: _path_cache_lock.release() Mike Looijmans Philips Natlab / Topic Automation Hi, Your optimisation is called double-checked locking, and it is now known to be broken, especially on multiple-CPU systems. The problem exists in numerous languages. Here is an explanation of the problem : http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html Now, maybe Python is not touched by this problem due to its relatively poor support for multithreading. The presence of the GIL guarantees that only one thread of Python bytecode interpreter runs at a time, but still, in a multiple CPU environment, you can get bitten by local caching issues. As this thing is a bit hairy, I think we should first strive for correctness, then care about performance later. And no, I won't bother you with one more quote about premature this and the root of that ;). Regards, Nicolas
Fwd: [mod_python] mod_python on Mobile phone - Nokia / Symbian S60 article
I sent this to Johan Wikman through the link available on the Nokia site : 8--8-- Hi Johan, I'm on the mod_python development team, and someone from the mailing list told us about your incredible port. Could you give us a bit of information on the subject, like what is the version you ported (we're currently releasing the 3.2.6 with a bunch of bugfixes), any difficulty you encountered, things you think we could improve to make your life easier, or things you might want to contribute to the project ? We would also liketo know if we could talk about your work when we make the announcement about the new release - something along the lines of mod_python is so portable it can even run on your favorite mobile phone thanks to the work of Nokia and so on and so forth (we love to boast like that ;). I cannot sign anything on behalf of the Apache Foundation, so that's only a prospective question, not a binding one ;). In any case, kudos for your work ! (I cc this to mod_python developers mailing list) 8--8-- -- Forwarded message -- From: Nicolas Lehuen [EMAIL PROTECTED] Date: 24 janv. 2006 20:50 Subject: Re: [mod_python] mod_python on Mobile phone - Nokia / Symbian S60 article To: David Fraser [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] I'm contacting Jonathan Wikman through the link available on the site. 2006/1/24, Nicolas Lehuen [EMAIL PROTECTED]: Incredible ! That would be nice to have a but of information about this, like the version they used and any caveat they noticed. This could be a great piece of news to generate buzz around mod_python, something that could be announced with the latest 3.2 release ;). Regards, Nicolas 2006/1/24, David Fraser [EMAIL PROTECTED]: Hi all This article is really interesting: http://research.nokia.com/research/software/mobile-web-server/index.html They seem to have ported Apache and mod_python to the Symbian platform... Has there been anything about this on the lists or did I miss it? David ___ Mod_python mailing list [EMAIL PROTECTED] http://mailman.modpython.org/mailman/listinfo/mod_python
Fwd: mod_python 3.2.6 (Final!) available for testing
Mike, I forward your +1 to python-dev since you only sent it to me. Regards, Nicolas -- Forwarded message -- From: Mike Looijmans [EMAIL PROTECTED] Date: 18 janv. 2006 08:21 Subject: Re: mod_python 3.2.6 (Final!) available for testing To: [EMAIL PROTECTED] +1 Windows XP Pro SP2 Apache 2.0.54 Python 2.4.2 -- Mike Looijmans Philips Natlab / Topic Automation Nicolas Lehuen wrote: You can fetch the Win32 version for Python 2.3 and Python 2.4 here : http://nicolas.lehuen.com/download/mod_python/
Re: please set up a mod_python core group
Hi, It's OK for me ! Regards, Nicolas 2006/1/19, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Thanks Roy. Very timely, since 3.2.6 is (so far) going to be a final/stable release. I propose that for starters those people are: me (I'm also in the Apache HTTP Server PMC) Jim Gallacher Nicolas Lehuen Graham Dumpleton Just to clarify this a bit - I think a +1 on successful test for a particular OS/whatever combination from any of the above people is NOT the same as the binding +1 Roy's referring to. So when we're done collecting +1's which are just test results from subscribers of the list (and any subscriber can send a +1), then at least 3 of the above list need to agree that we have sufficient approval to go ahead with the release. Roy - could you confirm this makes sense? Grisha On Wed, 18 Jan 2006, Roy T. Fielding wrote: It looks like mod_python is making good progress and everyone is collaborating in the Apache way of testing and voting. That's great! Unfortunately, I have almost no insight into who these great people are that are doing the RM task and testing and voting and preparing for a next release. That's not so great, since it is my job (as VP of Apache HTTP Server Project) to be sure that the ASF knows all this work is being done in its name and so that all of the people doing it are appropriately recognized for their work. So, please, take a few moments to decide amongst yourselves who should have binding votes on mod_python (i.e., who has earned it), keeping in mind that you need at least three binding +1 votes in order to make any release at Apache, and send me a list of names and email addresses of those people so that I can properly record them in our records. Cheers, Roy T. Fieldinghttp://roy.gbiv.com/ for the Apache HTTP Server PMC
Re: mod_python 3.2.6 (Final!) available for testing
You can fetch the Win32 version for Python 2.3 and Python 2.4 here : http://nicolas.lehuen.com/download/mod_python/ I have successfully tested and give my +1 for : Windows 2000 Server SP4, Python 2.3 Windows XP Pro SP2, Python 2.4 Regards, Nicolas 2006/1/16, Jim Gallacher [EMAIL PROTECTED]: Good news everyone! I made a mistake in tagging 3.2.6 as beta instead of final. The new tarball is now available for testing. This is the same code as yesterday's 3.2.6b.tgz but with the correct version information. This is the one we've all been waiting for! :) Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Ga
[jira] Updated: (MODPYTHON-93) Improve util.FieldStorage efficiency
[ http://issues.apache.org/jira/browse/MODPYTHON-93?page=all ] Nicolas Lehuen updated MODPYTHON-93: Fix Version: 3.3 Version: 3.2 (was: 3.3) Improve util.FieldStorage efficiency Key: MODPYTHON-93 URL: http://issues.apache.org/jira/browse/MODPYTHON-93 Project: mod_python Type: Improvement Components: core Versions: 3.2 Reporter: Jim Gallacher Assignee: Jim Gallacher Priority: Minor Fix For: 3.3 Attachments: modpython325_util_py_dict.patch Form fields are saved as a list in a FieldStorage class instance. The class implements a __getitem__ method to provide dict-like behaviour. This method iterates over the complete list for every call to __getitem__. Applications that need to access all the fields when processing the form will show O(n^2) behaviour where n == the number of form fields. This overhead could be avoided by creating a dict (to use as an index) when the FieldStorage instance is created. Mike Looijmans has been investigating StringField and Field as well. It is probably reasonable to include information on his work in this issue as well, so that we can consider all of these efficiency issues in toto. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: 3.2.6b
Hi Grisha, Having a look at the bug list I don't see anything that should prevent us from releasing the 3.2 version. There doesn't seem to be any bug due to some regression, all the new bugs were also found in 3.1. So I think we won't disappoint anyone ! I truly think we should not try to be perfect, and make this long due release of the 3.2 version. We'll have plenty of time later on to fix the next batch of bugs. Regards, Nicolas 2005/12/28, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I hope everyone's having a merry Christmas or whatever other holidays you're celebrating :-) I haven't been following the list very closely lately, but it looks like last time 3.2.6b was brought up a bunch of bugs came out of the woodwork. Does it look like we're near being ready to roll 3.2.6b now? Grisha
[jira] Updated: (MODPYTHON-104) Allow Python code callouts with mod_include (SSI).
[ http://issues.apache.org/jira/browse/MODPYTHON-104?page=all ]
Nicolas Lehuen updated MODPYTHON-104:
-
Fix Version: 3.3
Allow Python code callouts with mod_include (SSI).
--
Key: MODPYTHON-104
URL: http://issues.apache.org/jira/browse/MODPYTHON-104
Project: mod_python
Type: New Feature
Components: core
Reporter: Graham Dumpleton
Fix For: 3.3
The mod_include module supporting server side includes (SSI), provides a
means of registering new element tags which trigger callouts to other code in
separate Apache modules. This is used for example in mod_perl to allow Perl
language code to be used with server side includes:
!--#perl sub=MySSI::remote_host --
!--#perl arg=Hello arg=SSI arg=World
sub=sub {
my($r, @args) = @_;
print qq(@args);
}
--
An equivalent feature for Python was previously asked about on the mailing
list back in 2004:
http://www.modpython.org/pipermail/mod_python/2004-January/014832.html
Since it seems entirely reasonable that such integration of mod_python and
mod_include would be possible, thought it would be good to log it as a
possible new feature.
Because of SSI's support for basic conditionals, includes and other callout
mechanisms, would be a good quick and dirty way of doing templating without
having to resort to PSP, or other high level templating systems.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: 3.2.6b
2005/12/28, Graham Dumpleton [EMAIL PROTECTED]: Grisha, I have though asked for a small change to be made which will allow me to make available a proposed new module importer when its done and for people be able to trial it without having to patch their source code. See: http://www.mail-archive.com/python-dev@httpd.apache.org/msg00904.html for details of what I was proposing. Would appreciate it if you can indicate either way whether you are okay with this small change being made or not. Graham I've tested your patch and of course it doesn't break anything in our unit tests. I can't see how it can break anything, therefore I've checked it in in the hope that it will make it for 3.2.6 final. Grisha, tell me if you disagree with that, I'll roll it back at once. Regards, Nicolas
[jira] Resolved: (MODPYTHON-97) mod_python.publisher iterables and content_type broken
[ http://issues.apache.org/jira/browse/MODPYTHON-97?page=all ]
Nicolas Lehuen resolved MODPYTHON-97:
-
Fix Version: 3.2
Resolution: Fixed
Assign To: Nicolas Lehuen
Fixed this by reverting the changes from MODPYTHON-15.
mod_python.publisher iterables and content_type broken
--
Key: MODPYTHON-97
URL: http://issues.apache.org/jira/browse/MODPYTHON-97
Project: mod_python
Type: Bug
Components: publisher
Versions: 3.2
Reporter: Graham Dumpleton
Assignee: Nicolas Lehuen
Fix For: 3.2
In 3.2, mod_python.publisher was modified so that if it encountered an
interable it would recursively iterate over the items and publish each with
the result being concatenated.
FWIW, I personally didn't like this as I saw it potentially changing the
behaviour of existing code, although perhaps in contrived cases or for test
code only. I saw that this sort of behaviour should have been managed by the
user by explicit use of a wrapper class instead, rather than it being magic.
End of ramble. :-)
Regardless of my concerns, the behaviour that was added is broken.
Specifically, mod_python.publisher is setting the content type based on the
content of the first item returned from the iterable. For example, consider
the following:
index = [
'htmlbodyp',
1000 * X,
'/p/body/html',
]
When published, this is resulting in the content type being set to
'text/plain' and not 'text/html'. In part this is perhaps caused by the fact
that the content type check is now performed by looking for a trailing
'/html' in the content whereas previously it would look for a leading
'html'. This was changed because all the HTML prologue that can appear
before 'html' would often throw out this check with the HTML not being
automatically being detected. Thus at the time it was thought that looking
for the trailing '/html' would be more reliable. It ain't going to help to
go back to using a leading 'html' check though as the first item may only
contain the prologue and not 'html'.
These checks are only going to work for iterables if the results of
publishing of each item were added to the end of a list of strings, rather
than being written back immediately using req.write(). Once all that has been
returned by the iterable is obtained, this can all be joined back together
and then the HTML check done.
Joining all the separate items returned from the iterable back together
defeats the purpose of what this feature was about in the first place and may
result in huge in memory objects needing to be created to hold the combined
result just so the HTML check can be done.
The only way to avoid the problem is for the content type to be set
explicitly by the user before the iterable is processed. This is a bit tricky
as it is mod_python.publisher which is automagically doing this. The best you
can do is something like:
class SetContentType:
def __init__(self,content_type):
self.__content_type = content_type
def __call__(self,req):
req.content_type = self.__content_type
return
index = [
SetContentType('text/html'),
'htmlbodyp',
1000 * X,
'/p/body/html',
]
Once you start doing this, the user may as well have provided their own
published function in the first place that set the content type and manually
iterated over items and wrote them to req.write(). This could also be managed
by a user specified wrapper class which is how I saw this as preferably being
done in the first place. Ie.,
class PublishIterable:
def __init__(self,value,content_type):
self.__value = value
self.__content_type = content_type
def __call__(self,req):
req.content_type = self.__content_type
for item in self.__value:
req.write(item)
_values = [
'htmlbodyp',
1000 * X,
'/p/body/html',
]
index = PublishIterable(_values,'text/html')
Personally I believe this automagic publishing of iterables should be removed
from mod_python.publisher. You might still provide a special function/wrapper
that works like PublisherIterable but handles recursive structures and
callable objects in the process, but I feel it should be up to the user to
make a conscious effort to use it and mod_python.publisher shouldn't assume
that it should process any iterable in this way automatically.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Reopened: (MODPYTHON-15) Publisher : iterable return values should be corretly published
[ http://issues.apache.org/jira/browse/MODPYTHON-15?page=all ]
Nicolas Lehuen reopened MODPYTHON-15:
-
Reopened to fix MODPYTHON-97.
Publisher : iterable return values should be corretly published
---
Key: MODPYTHON-15
URL: http://issues.apache.org/jira/browse/MODPYTHON-15
Project: mod_python
Type: Improvement
Versions: 3.1.3
Reporter: Nicolas Lehuen
Assignee: Nicolas Lehuen
Priority: Minor
Fix For: 3.2
Suppose this function in a published module :
def index(req)
req.content_type = 'text/plain'
yield '1\n'
yield '2\n'
yield '3\n'
yield '4\n'
When published, this module should return a text content with '1\n2\n3\n4\n'.
This could also be useful with a file() object, since they are iterable ;
this would provide another way to send a file, only slightly less performing
than the send_file() method. Handy when you want to filter a file :
def filter(req,filename):
f = open(filename,'r')
for line in f:
yield re.sub('foo','bar',line)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-99) accessing some request or server object members causes a segfault
[ http://issues.apache.org/jira/browse/MODPYTHON-99?page=comments#action_12360041 ] Nicolas Lehuen commented on MODPYTHON-99: - Modifying util.c so that tuple_from_array_header and tuple_from_method_list return an empty tuple instead of None fixes the problem, but I still have t(o check whether this change won't break anything else. accessing some request or server object members causes a segfault - Key: MODPYTHON-99 URL: http://issues.apache.org/jira/browse/MODPYTHON-99 Project: mod_python Type: Bug Components: core Versions: 3.2 Reporter: Jim Gallacher Priority: Critical Attachments: md-20051209.diff Martin Devara discovered a segfault when accessing some request object members. For example: def handler(req): req.content_type = text/plain req.write(EE\n) a = getattr(req,allowed_methods); return apache.OK Futher investigation revealed problems with several getter functions in requestobject.c and serverobject.c. The root of the problem seems to be pointer dereferencing errors in the getter code. The affected functions and the members which use them are: src/requestobject.c getreq_rec_ml allowed_methods getreq_rec_ah content_languages allowed_xmethods src/serverobject.c getsrv_recmbr_ah names wild_names Martin has provided a patch to fix the bug. (Thanks to Martin for tracking this down and providing the fix.) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-99) accessing some request or server object members causes a segfault
[ http://issues.apache.org/jira/browse/MODPYTHON-99?page=comments#action_12360042 ] Nicolas Lehuen commented on MODPYTHON-99: - OK, if we modify tuple_from_array_header and tuple_from_method_list, here are the members that would be defined as an empty tuple rather than a None value when the underlying Apache data is not defined : server.names server.wildnames req.allowed_xmethods req.allowed_methods req.content_languages I don't know if we should change the behaviour of those members or simply alter the unit tests, though. accessing some request or server object members causes a segfault - Key: MODPYTHON-99 URL: http://issues.apache.org/jira/browse/MODPYTHON-99 Project: mod_python Type: Bug Components: core Versions: 3.2 Reporter: Jim Gallacher Priority: Critical Attachments: md-20051209.diff Martin Devara discovered a segfault when accessing some request object members. For example: def handler(req): req.content_type = text/plain req.write(EE\n) a = getattr(req,allowed_methods); return apache.OK Futher investigation revealed problems with several getter functions in requestobject.c and serverobject.c. The root of the problem seems to be pointer dereferencing errors in the getter code. The affected functions and the members which use them are: src/requestobject.c getreq_rec_ml allowed_methods getreq_rec_ah content_languages allowed_xmethods src/serverobject.c getsrv_recmbr_ah names wild_names Martin has provided a patch to fix the bug. (Thanks to Martin for tracking this down and providing the fix.) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: What's in a URL ?
2005/12/5, David Fraser [EMAIL PROTECTED]: Nicolas Lehuen wrote: As for the colophon : I initially built this chart on Excel 2003, then feeling a bit guilty, I decided to switch to OpenOffice 2 (developer release). I have then discovered that OpenOffice is far less intuitive in the domain of merged cells or cell borders. For example, you cannot insert a line on a sheet in the middle of a merged cell (Excel allows this). You have to split the merged cell, insert the line, and merge again, discovering that doing so has broke your cell borders, so you have to set them again. That's where you start to regret Excel's border drawing tool... Anyway, as a result, the borders may look a bit funny. I think we should switch to HTML as soon as the chart is stabilized, but until then using a spreadsheet makes editing the chart somewhat easier.You did report a bug to OpenOffice.org didn't you? :-)DavidIt's already been reported :http://qa.openoffice.org/issues/show_bug.cgi?id=14769 Regards,Nicolas
Re: What's in a URL ?
2005/12/5, Nicolas Lehuen [EMAIL PROTECTED]: 2005/12/5, David Fraser [EMAIL PROTECTED]: Nicolas Lehuen wrote: As for the colophon : I initially built this chart on Excel 2003, then feeling a bit guilty, I decided to switch to OpenOffice 2 (developer release). I have then discovered that OpenOffice is far less intuitive in the domain of merged cells or cell borders. For example, you cannot insert a line on a sheet in the middle of a merged cell (Excel allows this). You have to split the merged cell, insert the line, and merge again, discovering that doing so has broke your cell borders, so you have to set them again. That's where you start to regret Excel's border drawing tool... Anyway, as a result, the borders may look a bit funny. I think we should switch to HTML as soon as the chart is stabilized, but until then using a spreadsheet makes editing the chart somewhat easier.You did report a bug to OpenOffice.org didn't you? :-)DavidIt's already been reported : http://qa.openoffice.org/issues/show_bug.cgi?id=14769 Regards,Nicolas ... since 2003. Duh.
Re: Testing mod_python on win32
My bad... It seems it's not necessary to stop the Apache server. I was a bit confused by the Apache Monitor, a Win32 application putting an icon in the tray area showing the state of the Apache server and allowing you to control it. Turns out the monitor is a bit messed up by the test procedure, showing the status of the test server and not the official server. Thus when the tests stop, the monitor shows that the Apache server is stopped even though the official one isn't. I have changed the documentation accordingly.Regards,Nicolas2005/12/5, Graham Dumpleton [EMAIL PROTECTED]: I'm a bit confused by: - The only trick is that you'll have to stop your Apache server before launching the test, as the start/stop command can only apply to one singleApache instance.Does this apply to UNIX as well as Win32?I ask as I have never bothered to explicitly shut down any running instance ofApache, yet haven't noticed any problems with running the tests. Ifthis is a Win32specific instruction, you might want to note it as such. On UNIXsystems, wherethe web server may be doing real work, people may not want to shut it down justto be able to test a new separate version of mod_python that hasn'tbeen installedyet.GrahamOn 06/12/2005, at 8:02 AM, Nicolas Lehuen wrote: Hi David, To follow my old promise, I've just checked in a bit of documentation on how to run the test suite, including on Win32. I've also added a few self-test in the test module, so that the most obvious setup mistakes are notified to the user. Here is the documentation, directly from the Subversion repository : http://svn.apache.org/repos/asf/httpd/mod_python/trunk/test/README This should eventually be converted to TeX and integrated into the real documentation, but for various reasons this way is the quickest way to put it online. It's much better than the previous README file anyway (it was basically saying keep out unless you know what you're doing ;). Hope this helps. Regards, Nicolas 2005/12/5, David Fraser [EMAIL PROTECTED]: As afar as I can recall, Nicolas Lehuen is the only guy who's been able to run the tests on win32 Has anybody else been able to? Can we put together some hints as to how to do it? David
[jira] Updated: (MODPYTHON-78) No support for Apache 2.2 yet
[ http://issues.apache.org/jira/browse/MODPYTHON-78?page=all ] Nicolas Lehuen updated MODPYTHON-78: Summary: No support for Apache 2.2 yet (was: No support for Apache 2.1 yet) Now that Apache 2.2 is out, and mod_python 3.2 close to release, maybe it's time to have a look at this for mod_python 3.3 ? No support for Apache 2.2 yet - Key: MODPYTHON-78 URL: http://issues.apache.org/jira/browse/MODPYTHON-78 Project: mod_python Type: Bug Components: core Versions: 3.2 Reporter: Nicolas Lehuen Fix For: 3.3 See http://article.gmane.org/gmane.comp.apache.mod-python.devel/1425 for some remarks by Jorey Bump, raised during the 3.2.1-BETA tests. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: What's in a URL ?
Hi Jim,1. I chose the colours to aid in reading, but I tried to regroup items logically. For example I chose a weird orange for environment variables. Anyway, I'm thinking that I could use colors to represent the dependencies (what data comes from the client, what data comes from the server, and after this dichotomie, a set of colors for Apache-level APIs, CGI env-vars, and mod_python). That's a work in progress... 2. I don't know - I did not made this to distribute, just to fuel our discussion about the various way to get information about the request, and the mess it can cause. But if you guys find it worth publishing, why not. Regarding 2c, I solved the problem by dropping OpenOffice and doing it in HTML (I exported from OO to HTML and cleaned up the mess manually). I've checked in the result in the Doc directory.Regards,Nicolas 2005/12/3, Jim Gallacher [EMAIL PROTECTED]: Nicolas Lehuen wrote: Hi, Following last week's discussion about the various parts composing an URL and how to get them from Apache and/or mod_python, here is my first try at a chart that sums up what we know. It show a sample URL and how different components of the application server see it or contribute to it.Interesting view. Couple of questions:1. Any significance to the colours or are they just to aid in reading?2. How do you envisage this being distributed? a. On the mod_python website?b. Once it's complete, rewrite it in LateX so it's integrated withthe generated html-docs?c. Bundle with html-docs but generate the file (html or pdf) fromthe ods source? From the perspective of creating the releases 2.b is likely best, butmaking this kind of table in LaTeX goes *way* beyond my skills.If you see us using 2.c then we need to think about how to automate openoffice to create the file during the packaging.Jim
Re: Apache 2.2 released - apparently breaks the mod API compatibility with 2.0
I'll have to wait for the Win32 source code tree to be released to build it and test your patch. Hopefully it'll be out soon.Is there a wait to use macro directives so that we don't need to maintain two separate branches ? A define that we could pass when building mod_python to select the Apache version we're building against, maybe ?
Regards,Nicolas2005/12/3, Jorey Bump [EMAIL PROTECTED]:
Nicolas Lehuen wrote: http://httpd.apache.org/download.cgi Apache 2.2 add-in modules are not compatible with Apache 2.0 or 1.3 modules. If you are running third party add-in modules, you will need to
obtain new modules written for Apache 2.2 from that third party before you attempt to upgrade from Apache 2.0. Great, now we're having to support three separate version : mod_python
2.7 for Apache 1.3.x (though it's a bit unsupported now, isn't it ?), mod_python 3.2 for Apache 2.0.x and mod_python 3.2.x for Apache 2.2... It's not a big surprise, though, since we already have this issue :
http://issues.apache.org/jira/browse/MODPYTHON-78 Does anyone knows anything about the API changes ?I've attached a source tree patch against
3.2.5b that will work withapache 2.2.0. It still fails one test in the test suite, but seems toload fine in apache and run modules in Publisher.To apply the patch, move into the source code directory and issue the
following command: patch -p1 /path/to/mod_python-3.2.5b.patchSorry, I don't do Apache on Windows. Could someone follow up withinstructions for that platform (beyond install Cygwin)? :)
Here are some key points:APR_STATUS_IS_SUCCESS is gone.apr_sockaddr_port_get is gone.mod_auth is now mod_auth_basic.auth_module is now auth_basic_module.Affected files are:src/connobject.c
src/filterobject.ctest/test.pyTo fix the APR_STATUS_IS_SUCCESS issue, I deleted the code that used it,without replacement. That may be suboptimal, if the code serves a usefulpurpose. :)
To fix the apr_sockaddr_port_get issue, I restored makesockaddr fromconnobject.c in 3.2.1b. This was obviously replaced for a reason inlater versions, with the unfortunate choice of a deprecated functionfrom the API. The original issue needs to be revisited to determine a
more compatible solution.I'm unable to diagnose the remaining failure in the test suite: * Testing internally (status messages go to error_log)F==
FAIL: test_internal (__main__.PerRequestTestCase)--Traceback (most recent call last): File test.py, line 1249, in test_internal
self.fail(Some tests failed, see error_log)AssertionError: Some tests failed, see error_log--Ran 43 tests in 61.161s
FAILED (failures=1)FStopping Apache.../usr/local/apache2.2.0/bin/httpd -k stop -f/home/jorey/src/mod_python-3.2.5b/test/conf/test.conf==
FAIL: testPerRequestTests (__main__.PerInstanceTestCase)--Traceback (most recent call last): File test.py, line 1805, in testPerRequestTests
self.failUnless(result.wasSuccessful())AssertionError--Ran 6 tests in 107.536sFAILED (failures=1)The error log includes this line at the end:
logs/error_log:[Sat Dec 03 15:31:15 2005] [error] [client 127.0.0.1]..F.\n==\nFAIL:test_server_members
(tests.SimpleTestCase)\n--\nTraceback(most recent call last):\nFile/home/jorey/src/mod_python-3.2.5b/test/htdocs/tests.py, line 446, in
test_server_members\nself.fail(server.keep_alive_timeout should be15.0)\nAssertionError: server.keep_alive_timeout should be15.0\n\n--\nRan
8 tests in 0.336s\n\nFAILED (failures=1)\ndiff -uNr mod_python-3.2.5b/src/connobject.c mod_python-3.2.5b.new/src/connobject.c--- mod_python-3.2.5b/src/connobject.c2005-11-12 13:59:35.0
-0500+++ mod_python-3.2.5b.new/src/connobject.c2005-12-03 15:26:27.0 -0500@@ -78,12 +78,6 @@ rc = ap_get_brigade(c-input_filters, bb, mode, APR_BLOCK_READ, bufsize); Py_END_ALLOW_THREADS;
-if (! APR_STATUS_IS_SUCCESS(rc)) {-PyErr_SetObject(PyExc_IOError,-PyString_FromString(Connection read error));-return NULL;-}- /*
* loop through the brigade reading buckets into the string*/@@ -312,24 +306,17 @@***utility func to make a socket address*/- static PyObject *makesockaddr(struct apr_sockaddr_t *addr)
-{+{ PyObject *addrobj = makeipaddr(addr); PyObject *ret = NULL; if (addrobj) {-apr_port_t port;-if(apr_sockaddr_port_get(port, addr)==APR_SUCCESS) {-ret = Py_BuildValue(Oi, addrobj, port );
-}-else {-PyErr_SetString(PyExc_SystemError,apr_sockaddr_port_get failure);-}+ret = Py_BuildValue(Oi, addrobj
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
2005/11/29, Nicolas Lehuen [EMAIL PROTECTED]: 2005/11/29, Mike Looijmans [EMAIL PROTECTED]: Nicolas Lehuen wrote: Why is the ordering so important ? I do understand we need to support multiple values per field name, but I don't see why ordering is needed. Because there are applications out there that will break if you change it. Besides that, think about a form with a few text entry boxes (all the same name, e.g. spouse). It would be very confusing for a user of that page to see the text re-ordered every time he clicks one of the buttons on the page. (I'm perfectly aware of at least 4 alternatives, but that is not my point here). From the page code I've written and seen so far, the order of differently named fields is not important. I haven't seen a case where a form expecting a=1b=2 would fail if you pass it b=2a=1. But I have seen cases where a=1x=2x=3 is not the same as a=1x=3x=2. The simple dictionary implementation as proposed would not break that code. -- Mike Looijmans Philips Natlab / Topic Automation Hi Mike, As Jim pointed out, even if using a simple dict structure would not enable us to preserve the *key* ordering, it would still allow us to preserve the *value* ordering for fields with the same key, since they would be added to lists in the same order the browser would send them. So my guess is that preserving key order is not required, but preserving value order for a given key is. In that case a simple dict with list values is sufficient, easy to implement and efficient. Your examples are easily handled with this solution. I think we should check how this problem has been solved in other programming environments. I'll check how this was done in the Java servlet API. Well, the Java Servlets 2.4 API doesn't say anything about field order (see in javax.servlet.ServletRequest.getParameterName() or page 35 in servlet-2_4-fr-spec.pdf). It turns out this problem was raised in the antique JServ project : http://archive.apache.org/gnats/5211 One proposal was to use an OrderedHashtable. As you can see, the reply was quite definite : No. There is nothing in the spec that says that these parameters should be in any sort of order. CGI scripts that expect them to be in order are coded incorrectly IMHO. Regards, Nicolas
Re: [mod_python] Re: next beta
OK, great ! I'm ready for the next beta, then, and this time I can produce Python 2.3 and Python 2.4 versions for Win32. Jim, please fire at will ! Regards, Nicolas 2005/11/14, Alexis Marrero [EMAIL PROTECTED]: Nicholas, Just finish testing with couple of hundred files and is working AS SEEN ON TV. Thanks for commenting the code. /amn On Nov 14, 2005, at 10:08 AM, Nicolas Lehuen wrote: Alexis, sorry but this is not the latest version. I've made some changes to your version with the hope of simplifying the source code and documenting it. Could you please try the version you'll find at : http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/lib/python/ mod_python/util.py?rev=332779view=markup Be sure to use the whole file, and not just read_to_boundary. Indeed, I've introduced some changes in FieldStorage.__init__ that will be required by read_to_boundary. Thanks, Nicolas 2005/11/14, Alexis Marrero [EMAIL PROTECTED]: I did try the code and worked fine for my test file set (~55000 files). Like I stated in a previous email I made some changes based on Mike's concern about performance. So my last version of the function was, I tested it tih my file set and it passed all my tests. def read_to_boundary_new(self, req, boundary, file): previous_delimiter = '' bound_length = len(boundary) while 1: line = req.readline(readBlockSize) if line[:bound_length] == boundary: break if line[-2:] == '\r\n': file.write(previous_delimiter + line[:-2]) previous_delimiter = '\r\n' elif line[-1:] == '\r': file.write(previous_delimiter + line[:-1]) previous_delimiter = '\r' elif line[-1:] == '\n': if len(line[:-1]) 0: file.write(previous_delimiter + line[:-1]) previous_delimiter = '\n' else: previous_delimiter += '\n' else: file.write(previous_delimiter + line) previous_delimiter = '' /amn On Nov 14, 2005, at 8:34 AM, Jim Gallacher wrote: It looks like Nicolas was very busy on the weekend cleaning up remaining issues. Has Alexis had a chance to test the new upload code? Are there any objections to creating a 3.2.5 beta today? Jim ___ Mod_python mailing list [EMAIL PROTECTED] http://mailman.modpython.org/mailman/listinfo/mod_python
Re: mod_python 3.2.5b available for testing
Thanks for the information, I'll add your patch to the test suite. Regards, Nicolas 2005/11/15, Barry Pederson [EMAIL PROTECTED]: I've got failures that seem to be caused by the tests themselves, but with a bit of tweaking they pass. FreeBSD 6.0 Apache 2.0.55 port built WITH_THREADS=1 Python 2.4.2 The error_log shows: -- [Mon Nov 14 19:38:15 2005] [notice] mod_python: Creating 8 session mutexes based on 256 max processes and 0 max threads. [Mon Nov 14 19:38:15 2005] [alert] (2)No such file or directory: getpwuid: couldn't determine user name from uid 4294967295, you probably need to modify the User directive [Mon Nov 14 19:38:15 2005] [notice] Apache/2.0.55 (FreeBSD) mod_python/3.2.5b Python/2.4.2 configured -- resuming normal operations [Mon Nov 14 19:38:15 2005] [info] Server built: Nov 12 2005 23:05:22 [Mon Nov 14 19:38:15 2005] [debug] prefork.c(956): AcceptMutex: flock (default: flock) [Mon Nov 14 19:38:15 2005] [alert] Child 9492 returned a Fatal error... Apache is exiting! [Mon Nov 14 19:38:15 2005] [emerg] (2)No such file or directory: Couldn't initialize cross-process lock in child [Mon Nov 14 19:38:15 2005] [emerg] (2)No such file or directory: Couldn't initialize cross-process lock in child Googling that last message comes up with a suggesting that you specify a User in the http config. With the attached patch, the tests run httpd with a User www directive, and pass. Barry --- mod_python-3.2.5b-old/test/httpdconf.py Tue Sep 13 15:35:57 2005 +++ mod_python-3.2.5b/test/httpdconf.py Mon Nov 14 19:43:07 2005 @@ -264,6 +264,10 @@ def __init__(self, val='Off'): Directive.__init__(self, self.__class__.__name__, val) +class User(Directive): +def __init__(self, val='www'): +Directive.__init__(self, self.__class__.__name__, val) + class VirtualHost(ContainerTag): def __init__(self, addr, *args): ContainerTag.__init__(self, self.__class__.__name__, addr, args) --- mod_python-3.2.5b-old/test/test.py Mon Nov 14 12:09:49 2005 +++ mod_python-3.2.5b/test/test.py Mon Nov 14 19:56:03 2005 @@ -229,6 +229,7 @@ IfModule(!mod_dir.c, LoadModule(dir_module %s % quoteIfSpace(os.path.join(modpath, mod_dir.so, +User(www), ServerRoot(SERVER_ROOT), ErrorLog(logs/error_log), LogLevel(debug),
[jira] Resolved: (MODPYTHON-89) Add new apache.exists_config_define() function.
[ http://issues.apache.org/jira/browse/MODPYTHON-89?page=all ] Nicolas Lehuen resolved MODPYTHON-89: - Fix Version: 3.2 Resolution: Fixed Assign To: Nicolas Lehuen I've added the function and a test case. Add new apache.exists_config_define() function. --- Key: MODPYTHON-89 URL: http://issues.apache.org/jira/browse/MODPYTHON-89 Project: mod_python Type: Improvement Components: core Versions: 3.3 Reporter: Graham Dumpleton Assignee: Nicolas Lehuen Priority: Minor Fix For: 3.2 Add new function: apache.exists_config_define() The intent is that this function would wrap the underlying Apache function: ap_exists_config_define() This function allows one to determine if certain Apache command line options were used. For example, if the '-DONE_PROCESS' command line option was used in explicitly starting httpd. if apache.exists_config_define(ONE_PROCESS): ... do something Exposing of this function would provide equivalent functionality to the IfDefine directive available in Apache configuration files. The availability of this option would allow special debugging code to consult whether Apache is run in one process mode and thus determine whether the debugging code should be able to be run in the first place. Eg. pdb support. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-89) Add new apache.exists_config_define() function.
[ http://issues.apache.org/jira/browse/MODPYTHON-89?page=comments#action_12357495 ] Nicolas Lehuen commented on MODPYTHON-89: - Added some documentation to modpython4.tex. Add new apache.exists_config_define() function. --- Key: MODPYTHON-89 URL: http://issues.apache.org/jira/browse/MODPYTHON-89 Project: mod_python Type: Improvement Components: core Versions: 3.3 Reporter: Graham Dumpleton Assignee: Nicolas Lehuen Priority: Minor Fix For: 3.2 Add new function: apache.exists_config_define() The intent is that this function would wrap the underlying Apache function: ap_exists_config_define() This function allows one to determine if certain Apache command line options were used. For example, if the '-DONE_PROCESS' command line option was used in explicitly starting httpd. if apache.exists_config_define(ONE_PROCESS): ... do something Exposing of this function would provide equivalent functionality to the IfDefine directive available in Apache configuration files. The availability of this option would allow special debugging code to consult whether Apache is run in one process mode and thus determine whether the debugging code should be able to be run in the first place. Eg. pdb support. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
Hi guys,
In the pure if it ain't tested, it ain't fixed fashion, I've added a
unit test for file upload to the test suite. It uploads a randomly
generated 1 MB file to the server, and check that the MD5 digest
returned by the server is correct. I could not reproduce Alexis' bug
report this way, however. But I then I added a test with the
UNIX-HATERS handbook file ugh.pdf, and bang, here comes the bug.
I've checked in both unit tests into subversion, so that you can play with them. I'm now going to test Alexis' fix.
Regards,
Nicolas2005/11/6, Alexis Marrero [EMAIL PROTECTED]:
I don't have a function that creates the files but the I can pointyou to a file that has the problem, ironically is Unix HatersHandbook :) Well, at least is not the Python HH
http://research.microsoft.com/~daniel/uhh-download.htmlIt's MD5 is 9e8c42be55aac825e7a34d448044d0fe. I don't know what itends up been after upload with read_to_boundary().When you use the function to copy the file you will see that the
digest will be e45979254297b0ece9c182a789d7966e.I have other 5 files out of 78546 files that I'm testing it againstthat have the same issues, coincidentally there are all PDF files.Here is the script that I was testing it with.
def read_to_boundary(self, req, boundary, file): ''' read from the request object line by line with a maximum size, until the new line starts with boundary ''' previous_delimiter = ''
while 1: line = req.readline(116) if line.startswith(boundary): break if line.endswith('\r\n'): file.write(previous_delimiter + line[:-2])
previous_delimiter = '\r\n' elif line.endswith('\r') or line.endswith('\n'): file.write(previous_delimiter + line[:-1]) previous_delimiter = line[-1:]
else: file.write(previous_delimiter + line) previous_delimiter = ''#f = file('Perl Bookshelf [4th Ed]/mre/final/ch06.pdf.new', 'a+')#f = file('Pages User Guide.app/Contents/Resources/Italian.lproj/
Pages Manuale Utente.pdf', 'a+')f = file('ugh.pdf.new', 'a+')f.write('\r\n--myboundary--\r\n')f.seek(0)o = file('test.bin', 'wb')read_to_boundary(None, f, '--myboundary', o)o.close()/amn
On Nov 6, 2005, at 11:58 AM, Jim Gallacher wrote: Alexis, I wanted to add that I'm testing your code. Alexis Marrero wrote: Let me know any comments on it and if you test it and fails please
also let me know. I don't have subversion account neither I don't know how to use it thus this email. You don't need an account to use subversion anonymously. Just install subversion and grab a mod_python working copy.
$ svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk trunk This will checkout a working copy into a new directory called
trunk onyour machine. All of the following commands assume you are working in trunk/. Make your changes in your working copy, and then create a diff with: $ svn diff lib/python/mod_python/util.py
your-patch.diff The other commands which you'll find immediately useful are: svn update- update your working copy from the repository svn status- shows status of changes in your working copy
svn -u status- shows status of your copy against the repository I've found Version Control with Subverion is an excellent resource and is available online.
http://svnbook.red-bean.com/en/1.1/index.html Jim
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
OK, it looks like Alexis' fix solves the problem with ugh.pdf without
breaking the other unit tests. So I think we can safely integrate his
patch. Shall I do it ?
Regards,
Nicolas2005/11/6, Nicolas Lehuen [EMAIL PROTECTED]:
Hi guys,
In the pure if it ain't tested, it ain't fixed fashion, I've added a
unit test for file upload to the test suite. It uploads a randomly
generated 1 MB file to the server, and check that the MD5 digest
returned by the server is correct. I could not reproduce Alexis' bug
report this way, however. But I then I added a test with the
UNIX-HATERS handbook file ugh.pdf, and bang, here comes the bug.
I've checked in both unit tests into subversion, so that you can play with them. I'm now going to test Alexis' fix.
Regards,
Nicolas2005/11/6, Alexis Marrero [EMAIL PROTECTED]:
I don't have a function that creates the files but the I can pointyou to a file that has the problem, ironically is Unix HatersHandbook :) Well, at least is not the Python HH
http://research.microsoft.com/~daniel/uhh-download.htmlIt's MD5 is 9e8c42be55aac825e7a34d448044d0fe. I don't know what itends up been after upload with read_to_boundary().When you use the function to copy the file you will see that the
digest will be e45979254297b0ece9c182a789d7966e.I have other 5 files out of 78546 files that I'm testing it againstthat have the same issues, coincidentally there are all PDF files.Here is the script that I was testing it with.
def read_to_boundary(self, req, boundary, file): ''' read from the request object line by line with a maximum size, until the new line starts with boundary ''' previous_delimiter = ''
while 1: line = req.readline(116) if line.startswith(boundary): break if line.endswith('\r\n'): file.write(previous_delimiter + line[:-2])
previous_delimiter = '\r\n' elif line.endswith('\r') or line.endswith('\n'): file.write(previous_delimiter + line[:-1]) previous_delimiter = line[-1:]
else: file.write(previous_delimiter + line) previous_delimiter = ''#f = file('Perl Bookshelf [4th Ed]/mre/final/ch06.pdf.new', 'a+')#f = file('Pages User Guide.app/Contents/Resources/Italian.lproj/
Pages Manuale Utente.pdf', 'a+')f = file('ugh.pdf.new', 'a+')f.write('\r\n--myboundary--\r\n')f.seek(0)o = file('test.bin', 'wb')read_to_boundary(None, f, '--myboundary', o)o.close()/amn
On Nov 6, 2005, at 11:58 AM, Jim Gallacher wrote: Alexis, I wanted to add that I'm testing your code. Alexis Marrero wrote: Let me know any comments on it and if you test it and fails please
also let me know. I don't have subversion account neither I don't know how to use it thus this email. You don't need an account to use subversion anonymously. Just
install subversion and grab a mod_python working copy.
$ svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk
trunk This will checkout a working copy into a new directory called
trunk onyour machine. All of the following commands assume you are working in trunk/. Make your changes in your working copy, and then create a diff with: $ svn diff lib/python/mod_python/util.py
your-patch.diff The other commands which you'll find immediately useful are: svn update- update your working copy from the repository svn status- shows status of changes in your working copy
svn -u status- shows status of your copy against the repository I've found Version Control with Subverion is an excellent resource and is available online.
http://svnbook.red-bean.com/en/1.1/index.html Jim
[jira] Reopened: (MODPYTHON-40) FieldStorage : don't stream file uploads to memory
[ http://issues.apache.org/jira/browse/MODPYTHON-40?page=all ] Nicolas Lehuen reopened MODPYTHON-40: - The fix has a bug - see http://www.modpython.org/pipermail/mod_python/2005-November/019468.html and the python-dev mailing list (GMane archive are not up to date, sorry). Alexis Marrero [EMAIL PROTECTED] has proposed a fix, inspired from what CherryPy does. I've added a few unit tests to the mix, with the help of Jim Gallacher who found a small file that could always break the file upload system. FieldStorage : don't stream file uploads to memory -- Key: MODPYTHON-40 URL: http://issues.apache.org/jira/browse/MODPYTHON-40 Project: mod_python Type: Bug Versions: 3.1.4 Reporter: Nicolas Lehuen Fix For: 3.2 In mod_python.py/util.py, line 169, we stream a file upload to disk only if its Content-Disposition header features a filename attribute. Otherwise, the file is streamed to memory, thus opening a potential DoS attack by uploading very large files. We should : 1) Always stream file upload to disk 2) Define a default maximum file size which could be overridable. 3) Allow for the user to specify in which directory file uploads should be made, with a default to a temporary directory / file. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-21) Enable linking to shared python library
[ http://issues.apache.org/jira/browse/MODPYTHON-21?page=all ] Nicolas Lehuen updated MODPYTHON-21: Component: core Enable linking to shared python library --- Key: MODPYTHON-21 URL: http://issues.apache.org/jira/browse/MODPYTHON-21 Project: mod_python Type: Wish Components: core Versions: 3.2, 2.7.10, 3.1.3 Environment: Slackware Linux-10.0 / Apache-2.0.53 / Python-2.3 Reporter: Damjan Georgievski Priority: Minor I'd suggest to tweak the ./configure script, so that it tries to link to the python dynamic library first, and only if it fails to link the static. The only change wuld be to first try to link without the '-L/usr/lib/python2.3/config' option. A dynamically linked mod_python would get the benefit of always beeing the same version with the installed Python when minor versions are upgraded on the system. I've done this always and I've never had problems. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-78) No support for Apache 2.1 yet
[ http://issues.apache.org/jira/browse/MODPYTHON-78?page=all ] Nicolas Lehuen updated MODPYTHON-78: Component: core No support for Apache 2.1 yet - Key: MODPYTHON-78 URL: http://issues.apache.org/jira/browse/MODPYTHON-78 Project: mod_python Type: Bug Components: core Versions: 3.2 Reporter: Nicolas Lehuen Fix For: 3.3 See http://article.gmane.org/gmane.comp.apache.mod-python.devel/1425 for some remarks by Jorey Bump, raised during the 3.2.1-BETA tests. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: glue between apache and python logging
Nic, there is something I need to understand before giving my advice on the subject. I'm not familiar with the logging API, can you tell me how you configure Python to use this logging implementation ? Looks like we have to manipulate configuration object, set up handlers and so on... If so I guess we'd have to add a word or two in the documentation on the best way to do this in the mod_python context... Regards, Nicolas 2005/10/19, Nic Ferrier [EMAIL PROTECTED]: Nic Ferrier wrote: I really think the argument about mod_python being minimal does not apply here. Nick said it - mod_python is glue between Python and apache - well, making Python logging work directly to Apache seems like important glue to me.Nick [EMAIL PROTECTED] writes: And again I renew my argument that there needs to be some kind of contrib archive that is probably separate from mod_python and unsupported in the core distribution.Maybe a wiki or code repository or something to support all the contributions like this.That way all these neat things can be available to people who are using mod_python, but not burden the development process of mod_python itself with issues concerning the constributions.I agree. But I don't think logging glue should go in it. logging glue should be in core mod_python because it is glue, not a feature builton top of Apache or Python, simply a link between the two worlds.If logging is just available in an addon then developers have stillgot the problem of deploying mod_python PLUS pretty much essential stuff from so and so archive as well as the app.Nic
Re: glue between apache and python logging
OK now this is totally weird, we've got a Nic, a Nick and a Nicolas in the thread, watch out for mass confusion !2005/10/19, Nicolas Lehuen [EMAIL PROTECTED]:Nic, there is something I need to understand before giving my advice on the subject. I'm not familiar with the logging API, can you tell me how you configure Python to use this logging implementation ? Looks like we have to manipulate configuration object, set up handlers and so on... If so I guess we'd have to add a word or two in the documentation on the best way to do this in the mod_python context... Regards, Nicolas 2005/10/19, Nic Ferrier [EMAIL PROTECTED]: Nic Ferrier wrote: I really think the argument about mod_python being minimal does not apply here. Nick said it - mod_python is glue between Python and apache - well, making Python logging work directly to Apache seems like important glue to me.Nick [EMAIL PROTECTED] writes: And again I renew my argument that there needs to be some kind of contrib archive that is probably separate from mod_python and unsupported in the core distribution.Maybe a wiki or code repository or something to support all the contributions like this.That way all these neat things can be available to people who are using mod_python, but not burden the development process of mod_python itself with issues concerning the constributions.I agree. But I don't think logging glue should go in it. logging glue should be in core mod_python because it is glue, not a feature builton top of Apache or Python, simply a link between the two worlds.If logging is just available in an addon then developers have still got the problem of deploying mod_python PLUS pretty much essential stuff from so and so archive as well as the app.Nic
Re: glue between apache and python logging
2005/10/19, Nic Ferrier [EMAIL PROTECTED]: Is everyone here called Nic[h]olas?Nicolas Lehuen [EMAIL PROTECTED] writes: Nic, there is something I need to understand before giving my advice on the subject. I'm not familiar with the logging API, can you tell me how you configure Python to use this logging implementation ? Looks like we have to manipulate configuration object, set up handlers and so on... If so I guess we'd have to add a word or two in the documentation on the best way to do this in the mod_python context...Hmmm... logging is actually very simple if you want it to be.All you really have to do to use logging from Python is this: import logging# then sometime laterlogger = logging.getLogger()logger.debug(a log message)lot's more things are possible and the logging API is veryconfigurable. From an Apache point of view you do need to configure the defaultlogger in your handler so that it has access to the apache vhostlogger:hdlr = apache_log_handler.ApacheLogHandler(http) logging.getLogger().addHandler(hdlr)once you've done that then any handler which descends from the defaulthandler (and that's the normal way to do things) will log to Apache.Nic In that case, setting up the logging handler should be done by the user, making sure that it is set up only once per interpreter, even in the context of a multi-threaded MPM. It's not a trivial thing ; looks like this is a job for PythonImport. So my advice is that if the set up is done by the user, then the logging handler should be considered as contrib. If you provide a way for mod_python to directly set up the handler, maybe this could be considered part of mod_python. Regards, Nicolas
[jira] Updated: (MODPYTHON-60) PythonOption directive causes memory leak
[ http://issues.apache.org/jira/browse/MODPYTHON-60?page=all ]
Nicolas Lehuen updated MODPYTHON-60:
Fix Version: (was: 3.3.0)
PythonOption directive causes memory leak
-
Key: MODPYTHON-60
URL: http://issues.apache.org/jira/browse/MODPYTHON-60
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4, 3.1.3, 3.2.0
Environment: Linux
Reporter: Jim Gallacher
Assignee: Jim Gallacher
Priority: Critical
Fix For: 3.2.0
This was previously reported on the mod_python mailing list. See
http://www.modpython.org/pipermail/mod_python/2004-April/015395.html
A memory leak results when there is a PythonOption directive in the apache
config file. Leak occurs when PythonOption is in either VirtualHost or
Directory section.
For each request, approx 25 bytes of memory is leaked per PythonOption
directive.
Methodolgy (using top to gauge memory usage, 100,000 requests per test case):
def handler(req):
req.content_type = 'text/plain'
req.write('PythonOption test\n')
return apache.OK
1. No PythonOption directives:
1.4 % MEM
2. 50 PythonOption directives:
11.3% MEM
3. 100 PythonOption directives:
25.4 % MEM
I know 50 or 100 PythonOptions is not likely in a production system, but it
clearly demonstrate the leak.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-75) Crash and memory leak in python_merge_config due to use of apr_table_overlap
[ http://issues.apache.org/jira/browse/MODPYTHON-75?page=all ]
Nicolas Lehuen updated MODPYTHON-75:
Fix Version: (was: 3.3.0)
Crash and memory leak in python_merge_config due to use of apr_table_overlap
-
Key: MODPYTHON-75
URL: http://issues.apache.org/jira/browse/MODPYTHON-75
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4, 3.1.3
Environment: all
Reporter: Boyan Boyadjiev
Assignee: Jim Gallacher
Fix For: 3.2.0
Couse:
Usage of apr_table_overlap() function in the python_merge_config() and the
way of handling the pools for resource allocation there. In general using
this function in python_merge_config() mismatches the pool designated for
global configuration items and the one that has to be used for request local
data. By using a global pool for local data two request may collide when
accessing this one which leads to the crash.
The solution which works fine for us is to implement and use a
custom_table_overlap function, which does apr_table_overlay and then
apr_table_compress (similar aproach as in mod_perl):
/*
code begin
*/
/**
** modpython_table_overlap
**
* Replaces the apr_table_overlap() function using a specific pool
* for the resulting table.
*/
static apr_table_t *modpython_table_overlap(apr_pool_t *p,
apr_table_t *current_table,
apr_table_t *new_table)
{
apr_table_t *merge = apr_table_overlay(p, current_table, new_table);
apr_table_compress(merge, APR_OVERLAP_TABLES_SET);
return merge;
}
/**
** python_merge_dir_config
**
*/
static void *python_merge_config(apr_pool_t *p, void *current_conf,
void *new_conf)
{
py_config *merged_conf =
(py_config *) apr_pcalloc(p, sizeof(py_config));
py_config *cc = (py_config *) current_conf;
py_config *nc = (py_config *) new_conf;
apr_hash_index_t *hi;
char *key;
apr_ssize_t klen;
hl_entry *hle;
/* we basically allow the local configuration to override global,
* by first copying current values and then new values on top
*/
/** create **/
merged_conf-hlists = apr_hash_make(p);
merged_conf-in_filters = apr_hash_make(p);
merged_conf-out_filters = apr_hash_make(p);
/** merge directives and options **/
merged_conf-directives = modpython_table_overlap(p, cc-directives,
nc-directives);
merged_conf-options = modpython_table_overlap(p, cc-options,
nc-options);
/** copy current **/
merged_conf-authoritative = cc-authoritative;
merged_conf-config_dir = apr_pstrdup(p, cc-config_dir);
for (hi = apr_hash_first(p, cc-hlists); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void **)key, klen, (void **)hle);
apr_hash_set(merged_conf-hlists, key, klen, (void *)hle);
}
for (hi = apr_hash_first(p, cc-in_filters); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void **)key, klen, (void **)hle);
apr_hash_set(merged_conf-in_filters, key, klen, (void *)hle);
}
for (hi = apr_hash_first(p, cc-out_filters); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void **)key, klen, (void **)hle);
apr_hash_set(merged_conf-out_filters, key, klen, (void *)hle);
}
/** copy new **/
if (nc-authoritative != merged_conf-authoritative)
merged_conf-authoritative = nc-authoritative;
if (nc-config_dir)
merged_conf-config_dir = apr_pstrdup(p, nc-config_dir);
for (hi = apr_hash_first(p, nc-hlists); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void**)key, klen, (void **)hle);
apr_hash_set(merged_conf-hlists, key, klen, (void *)hle);
}
for (hi = apr_hash_first(p, nc-in_filters); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void**)key, klen, (void **)hle);
apr_hash_set(merged_conf-in_filters, key, klen, (void *)hle);
}
for (hi = apr_hash_first(p, nc-out_filters); hi; hi=apr_hash_next(hi)) {
apr_hash_this(hi, (const void**)key, klen, (void **)hle);
apr_hash_set(merged_conf-out_filters, key, klen, (void *)hle);
}
return (void *) merged_conf;
}
/*
code end
*/
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
OK, so on a non-threaded Apache, we can suppose we will be using the
prefork MPM, so we don't need any code to support threading in
mod_python, then, right ?
In this case instead of testing for WITH_THREAD in mod_python.c :
#ifdef WITH_THREAD
maybe we could test for WITH_THREAD and APR_HAS_THREADS :
#if APR_HAS_THREADS defined(WITH_THREAD)
Right ? This would remove all threading-related code from mod_python
when only prefork is available or when Python isn't compiled to support
threads (I which case I wonder how it works in a threaded Apache...).
I have given up using QEMU for now, minotaur is sufficient to make sure
mod_python builds on FreeBSD. Granted, it won't allow me to give any +1
since I cannot run the unit tests (or can I ?)...
Regards,
Nicolas
2005/9/11, Jim Gallacher [EMAIL PROTECTED]:
FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD.If
you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads.I'm also setting up FreeBSD under QEMU... so far so good, but installinganything using ports is really slow. QEMU's performance here is just
killing me. I guess I should have read the manual first and used thebinary packages for the software I wanted to install. :-(Regards,JimJim Gallacher wrote: Nicolas Lehuen wrote:
OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined.
Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in
the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1.
Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:
[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by
Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR
seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available,
which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all
version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer, but that will be long and troublesome. If someone has more clue on this
issue, feel free to tell us :). Regards, Nicolas 2005/9/10, Jim Gallacher [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:I'm stubling around in the dark here, but maybe this will create a spark
of an idea. I took a diff of mod_python.c from 3.1.4 and 3.2.1b andisolated the lines which correspond to the compilation error.Compiler messages
-mod_python.c:34: error: syntax error before '*' tokenmod_python.c:34: warning: type defaults to `int' in declaration of`interpreters_lock'
mod_python.c:34: warning: data definition has no type or storage classmod_python.c: In function `get_interpreter':mod_python.c:131: warning: implicit declaration of function
`apr_thread_mutex_lock'mod_python.c:161: warning: implicit declaration of function`apr_thread_mutex_unlock'mod_python.c: In function `python_init':
mod_python.c:517: warning: implicit declaration of function`apr_thread_mutex_create'mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (firstuse in this function)
Diff output---I've only copied the diff chunks which correspond to the complier errors
mentioned above.--- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005+++ mod_python-3.2.1b/src/mod_python.cTue Sep6 17:11:03 2005@@ -31,6 +31,8 @@
* (In a Python dictionary) */ static PyObject * interpreters = NULL;+static apr_thread_mutex_t* interpreters_lock = 0;+ apr_pool_t *child_init_pool = NULL;
... snip ...@@ -124,11 +128,15 @@ name = MAIN_INTERPRETER; #ifdef WITH_THREAD+apr_thread_mutex_lock(interpreters_lock);
PyEval_AcquireLock(); #endif... snip ...@@ -149,6 +158,7 @@ #ifdef WITH_THREAD
PyEval_ReleaseLock();+apr_thread_mutex_unlock(interpreters_lock); #endif... snip ...@@ -490,13 +506,15 @@
} /* initialize global Python interpreter if necessary */-if (! Py_IsInitialized())+if (initialized == 0 || !Py_IsInitialized())
{-+initialized = 1;+ /* initialze the interpreter */ Py_Initialize();
#ifdef WITH_THREAD+ apr_thread_mutex_create(interpreters_lock
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Well, it depends : #if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1 #else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0 #endif It's not only a matter of Python supporting threads, we must also have a thread-enabled APR. So that's the reason for the weird name I chose. But I don't mind changing it ! Regards, Nicolas2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD?GrishaOn Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] : FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2 . The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Duh, this is becoming difficult :) I was thinking that if APR_HAS_THREADS was 0, then Apache was forcibly ran in prefork mode, so there was no need for thread safety at all, given the fact that mod_python would only run one interpreter thread. So if WITH_THREAD was not defined, ORAPR_HAS_THREADS was 0, then we would not need any thread safety code. Hence the definition of MOD_PYTHON_WITH_THREAD_SUPPORT. You're right in writing that a user could launch a new thread in Python, provided that WITH_THREAD is defined, even if APR_HAS_THREADS==0. However, having a look at the parts of mod_python.c where the thread safety was put in, I think we can safely say that those parts are only called by mod_python (through python_handler, python_cleanup etc who call get_interpreter). Those parts are therefore always called in the same thread (if APR_HAS_THREADS==0, that is) and there is no need for thread synchronization to be done (no shared data between the main thread and the other user threads, no need to release the GIL etc.). BUT, I could be very, very wrong here, and your idea of reverting to a conservative shield python threading calls with WITH_THREAD and apr threading code with APR_HAS_THREADS is way more attractive to my tired mind right now. So if you want I can revert all this MOD_PYTHON_WITH_THREAD_SUPPORT hack. Regards, Nicolas2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: I'm not sure I understand this, perhaps someone could write a message tothe list explaining what we're doing here so there is a record. Sorry ifI'm being slow-headed here.To me it seems that when you use thread-related calls from Python, you wrap those in Python defines (WITH_THREAD) and when you use thread-relatedcalls from APR, you wrap those in APR defines (APR_HAS_THREAD), and that'sall?In other words - what does MOD_PYTHON_WITH_THREAD_SUPPORT accomplish that the above does not.Also, given:#if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1#else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0#endif Does this mean that if Python is compiled with thread support and APR isnot, MOD_PYTHON_WITH_THREAD_SUPPORT is 0 which means that the threadsafety code isn't there, but you still _can_ create threads because Python will let you - isn't this asking for a segfault/deadlock/whatever?GrishaOn Mon, 12 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Yes, this new code is something I commited on the 29/12/2004 (I used
the blame function of TortoiseSVN for that). It was a patch by Graham
to fix MODPYTHON-2.
The problem is not in the patch, but rather in the fact that APR seems
configured without the thread support while Python is configured with
thread support. mod_python.c assumes that is WITH_THREAD is defined,
then the APR mutex functions are available, which is wrong. Maybe we
should test for APR_HAS_THREADS instead ? In that case, won't this
cause any problems on threaded platforms ?
I don't know if this is a problem specific to minotaur or to all
version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and
I'll try using QEMU to run a FreeBSD setup on my computer, but that
will be long and troublesome. If someone has more clue on this issue,
feel free to tell us :).
Regards,
Nicolas2005/9/10, Jim Gallacher [EMAIL PROTECTED]: I'm stubling around in the dark here, but maybe this will create a spark of an idea. I took a diff of mod_python.c from
3.1.4 and 3.2.1b and isolated the lines which correspond to the compilation error. Compiler messages - mod_python.c:34: error: syntax error before '*' token
mod_python.c:34: warning: type defaults to `int' in declaration of `interpreters_lock' mod_python.c:34: warning: data definition has no type or storage class mod_python.c: In function `get_interpreter':
mod_python.c:131: warning: implicit declaration of function `apr_thread_mutex_lock' mod_python.c:161: warning: implicit declaration of function `apr_thread_mutex_unlock' mod_python.c: In function `python_init':
mod_python.c:517: warning: implicit declaration of function `apr_thread_mutex_create' mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (first use in this function)
Diff output --- I've only copied the diff chunks which correspond to the complier errors mentioned above. --- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005
+++ mod_python-3.2.1b/src/mod_python.cTue Sep6 17:11:03 2005 @@ -31,6 +31,8 @@* (In a Python dictionary) */ static PyObject * interpreters = NULL; +static apr_thread_mutex_t* interpreters_lock = 0;
+ apr_pool_t *child_init_pool = NULL; ... snip ... @@ -124,11 +128,15 @@ name = MAIN_INTERPRETER; #ifdef WITH_THREAD +apr_thread_mutex_lock(interpreters_lock);
PyEval_AcquireLock(); #endif ... snip ... @@ -149,6 +158,7 @@ #ifdef WITH_THREAD PyEval_ReleaseLock(); +apr_thread_mutex_unlock(interpreters_lock);
#endif ... snip ... @@ -490,13 +506,15 @@ } /* initialize global Python interpreter if necessary */ -if (! Py_IsInitialized())
+if (initialized == 0 || !Py_IsInitialized()) { - +initialized = 1; + /* initialze the interpreter */ Py_Initialize();
#ifdef WITH_THREAD + apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p); /* create and acquire the interpreter lock */ PyEval_InitThreads();
#endif So it would seem that the code causing the compile problems is new for 3.2. I also notice that in apr_arch_thread_mutex.h the typedef for apr_thread_mutex_t is wrapped by #if APR_HAS_THREADS / #endif.
Looking at the apache source in srclib/apr/locks/unix/thread_mutex.c, everything is also enclosed by #if APR_HAS_THREADS / #endif. eg, apr_thread_mutex_create, apr_thread_mutex_lock and
apr_thread_mutex_unlock. Hopefully this will give someone a clue as to what may be going on here with FreeBSD. Regards, Jim
Re: Getting ready for 3.2 beta 2
I've tried to build 3.1.4 from the tarball on minotaur and of course it works. Could it be possible that the recent changes in the configure script cause the problem ? Regards, Nicolas 2005/9/10, Jim Gallacher [EMAIL PROTECTED]: I thought I'd it a shot on minotaur as well. Poking around a bit reveals that the default apache is indeed 1.3. It looks like there might be a viable apache2 hiding in /usr/local/apache2-install/www.apache.org/current. eg ./configure --with-apxs=/usr/local/apache2-install/www.apache.org/current/apxs Unfortunately, I'm getting the same errors as Grisha reported starting with: mod_python.c:34: error: syntax error before '*' token Regards, Jim Nicolas Lehuen wrote: I tried to build it under minotaur as well, but ./configure only finds a 1.3.33 version of Apache, so I can't go further. I can't help much here since I'm not used to FreeBSD... Regards, Nicolas 2005/9/9, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just tried compiling it on minotaur and I get the same error. minotaur is FreeBSD 5.4, so it looks like we have a -1. I don't know how much time I'll have this weekend, so I might or might not look into the cause of this - but anyone else with access to a FreeBSD box, you're more than welcome to dig in... :-) Grisha On Fri, 9 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Don't know about versions, but I'd _really_ like to see a FreeBSD +1 at this point :-) Graham - don't you have FreeBSD access somewhere? If Graham can't help out maybe we could recruit a volunteer on the mod_python list? On the versioning discussion - I don't like 4.0, I think 3.3 should be the next version after 3.2.x. As far as even/odd stable/unstable - the Linux kernel folks have abandoned it because it didn't work for them. The fallacy is that you cannot know ahead of time what is stable and what is not. My preference is to just follow versions incrementally, and making it known which version is stable or not independently of the version number, which is what the HTTPD folks have been doing. I can't get worked up one way or another wrt to a version numbering scheme, as long as we release *something*. ;) Regards, Jim
Re: Possible Bug? Util.FieldStorage and the handling of content-type
Er, just as a notice, the second test for multipart/ was already correct, but I've changed it to 'not ctypes.startswith(multipart/)' for better code consistency. Regards, Nicolas 2005/9/8, Nicolas Lehuen [EMAIL PROTECTED]: Hi Dominic, That's perfectly acceptable. I've just used the startswith method of the str class instead of the lambda function you used. I've checked in the fix, hopefully it will appear in the next beta release. Regards, Nicolas 2005/9/8, Dominic Wong [EMAIL PROTECTED]: Hi, This was reported in 2003, and as far as I can tell the behaviour is still present in util.py: The full format of the media-type element according to RFC2616 is: snip media-type = type / subtype *( ; parameter ) type = token subtype = token /snip There are many user agents out there that use this extra parameter when posting data; Series 60 Symbian phones are the ones I'm concerned with atm, but I have also seen evidence of other browsers doing this as well. I believe that at the very least it should not assume that the entire Content-Type string is going to be either application/x-www-form-urlencoded or starting with multipart/, that it should be more tolerant towards parameters. I have supplied a patch for lib/python/mod_python/util.py against the 3.2.1 beta version; if someone could let me know if it is an acceptable solution or not, that would be great. Thanks! Dominic Wong James Baster wrote: Hey, Is this a bug? I found it while attempting to integrate worldpay into a solution for a client that uses Mod-Python. Basically, util.fieldstorage looks at the content-type header. If its a multi/*, it starts doing things with that. If it is application/x-www-form-urlencoded, it splits the POST parameters as it is meant to. Otherwise, the code raises an error. However, worldpay passes a content-type value of Content-Type: application/x-www-form-urlencoded; charset=ISO-8859-1. This causes problems, as util.fieldstorage refuses to recongise it. The solution I have temporarily put in place in our code is these two lines, directly before calling util.fieldstorage: t = split(request.headers_in[Content-Type], ;) request.headers_in[content-type] = t[0] I think this is safe, as there is no legitimate Content-type with a semi-colon character in it - so this fix should not break anything else. If it does, please let me know! However, my view is that util.fieldstorage should be able to handle this by itself. So anyway, I'm just reporting this issue for you to decide if it is a bug or not. Feel free to email with any comments ... Thanks, James. 116a117,118 startsWith = lambda string, start: string[:len(start)] == start 122c124 if ctype == application/x-www-form-urlencoded: --- if startsWith(ctype, application/x-www-form-urlencoded): 129c131 if ctype[:10] != multipart/: --- if not startsWith(ctype, multipart/):
