[jira] Commented: (MODPYTHON-223) Make sys.stdout/sys.stderr log properly to Apache error logs.
[ https://issues.apache.org/jira/browse/MODPYTHON-223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12495043 ] David Fraser commented on MODPYTHON-223: As a side comment that may be relevant, I've had print statements running under mod_python in Apache on Windows failing because the LOCALE is set to the C locale and so encoding things can cause issues... Make sys.stdout/sys.stderr log properly to Apache error logs. -- Key: MODPYTHON-223 URL: https://issues.apache.org/jira/browse/MODPYTHON-223 Project: mod_python Issue Type: New Feature Components: core Affects Versions: 3.3.1 Reporter: Graham Dumpleton When sys.stderr or sys.stdout are written to directly, or when the 'print' statement is used without a target file object, any text output will make it into the Apache error log files, but it will only get there after a flush has been performed on stdout/stderr. Most of the time the streams will only be flushed when Apache is being shutdown with the result being that the output is far removed in the log file from any other messages that may have been logged at the same time using the normal Apache error logging functions. Further, anything output via stdout/stderr does not have associated with it the error log prefix giving the time the output was generated. All this can be improved by replacing sys.stderr and sys.stdout with custom Python objects which buffer any output and flushing it through to the Apache error log using the proper ap_log_error() function after each newline is encountered. This will mean that output appears straight away as soon as newline sent and output will be prefixed with date/time string. Do note though that the output cannot be associated with a specific virtual host and thus would go to the main Apache error log file even if each virtual host has its own log file. This is the same as now though when sys.stderr and sys.stdout are used. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Core vote [Re: mod_python 3.3.0 beta available for testing]
I'm not core but I think its good practice to officially release this as a beta to the wider community before making it an actual release. I didn't test it because I was waiting for the core vote :-) David Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: core +1 on releasing it into the wild grisha I'm not sure what we're voting on here, and I'm not sure what I meant by the next level either. :) Is this a vote to give 3.3.0 beta a wider release (apache mirrors and so on), or a vote to go right to the 3.3.0 final release? I'm +1 either way. As I recall we didn't get much additional testing as a result of uploading the betas to the apache mirrors in the past. The people most likely to chip in with testing are already here on python-dev. Having the 3.2.x stable branch certainly facilitated releasing quick fixes for 3.2. There is no reason why we can't do the same with 3.3, so my inclination is to do an official 3.3.0 final release now rather than later. Jim On Mon, 11 Dec 2006, Jim Gallacher wrote: Test results so far, FYI. How long shall we wait until we kick it up to the next level? - jim +1 FreeBSD 6.1, Apache 2.2.3 (mpm-prefork), Python 2.4.3,1 +1 Linux Debian 3.1 Sarge, Apache 2.0.54 (mpm-prefork), Python 2.3.5 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.3.5 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.4.4 +1 Linux Debian Sid, Apache 2.2.3 (mpm-worker), Python 2.5 +1 Linux Fedora Core 5 (i386), Apache 2.2.2 (mpm-prefork), Python 2.4.3 +1 Linux Fedora Core 5 i386, Apache 2.2.2 (mpm-prefork), Python 2.4.3 +1 Linux Fedora Core 6, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Fedora Core 6 i386, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Fedora Core 6 x86_64, Apache 2.2.3 (mpm-prefork), Python 2.4.4 +1 Linux Slackware 10.2, Apache 2.2.3 (mpm-prefork), Python 2.4.1 +1 Linux Ubuntu 6.0.6, Apache 2.0.55 (mpm-worker), Python 2.4.3 +1 Linux Ubuntu 6.10, Apache 2.0.55 (mpm-prefork), Python 2.4.4c1 +1 Mac OS X (Darwin 8.8.1), Apache 2.2.3 (mpm-prefork), Python 2.4.2 +1 Mac OS X (PowerPC), Apache 2.2.1 (mpm-worker), Python 2.3.5 (OS Supplied Version) +1 Mac OS X (PowerPC), Apache 2.0.55 (mpm-worker), Python 2.3.5 (OS Supplied Version) +1 Microsoft Windows XP, Apache 2.0.58 (mpm_winnt), Python 2.5 +1 Microsoft Windows XP, Apache 2.0.58 (mpm_winnt), Python 2.4 +1 Microsoft Windows XP, Apache 2.2.2 (mpm_winnt), Python 2.5 +1 Microsoft Windows XP, Apache 2.2.2 (mpm_winnt), Python 2.4
Server Shutdown and register_cleanup
David Fraser wrote: Graham Dumpleton wrote: On 18/08/2006, at 7:48 PM, Richard Lewis wrote: On Tuesday 15 August 2006 23:31, Graham Dumpleton wrote: Richard Lewis wrote .. Do you mean that there will be no opportunity to have code run at server shutdown? Correct. Reason being that it doesn't actually work most of the time anyway. Because of how Apache is implemented, there is no reliable/safe way of implementing this feature. If one can't do it properly, it seems better not to attempt it at all. Goodness! So, in my mod_python applications I often acquire database handles and store them in objects outside of the handler() function so that they persist between requests. (This is to avoid the expense of acquiring a new handle for every request). And you can still do that. I then use a cleanup() function to release those database handles. The implication of this is that I will no longer be able to release the handles at server shutdown, yes? I think you miss what I have been saying. That it doesn't work now. If you were to put calls to apache.log_error() in your cleanup handler and shutdown Apache when it is handling a decent amount of traffic, you will probably find your cleanup functions aren't actually being called, or if they do, they might not exit and will hang at some point. What you are more likely to see in the Apache log is a series of SIGTERM signals being sent and then a SIGKILL signal which is forcibly killing off the process. Can anyone suggest an alternative method of doing this? Are there any Python tricks where you can execute code when the interpreter itself is about to stop? Python does have means of calling code on application shutdown, but because of how Apache uses signals to shutdown processes and how the Apache main loop works, with the main loop being managed by Apache and not Python they can't be used either. Or could I have a Python script running in another process which looks in to the mod_python process and periodically cleanly releases database handles? No. In the greater scheme of things it shouldn't ultimately matter. This is because your database server is going to notice that the connections to it have been dropped and will cleanup the resources on its side anyway. It has to do this as there is nothing to say that Apache or any client process connecting to it will not simply crash without cleanly disconnecting. In other words, you will not get resource leaks. That things didn't get cleanup in the Apache child process doesn't matter as it has been killed anyway, with all its memory being released back to operating system. So you are saying: 1) There is a mechanism for cleaning up code 2) This mechanism is not reliable 3) Since databases have to assume clients are not reliable, they clean up for them anyway 4) Therefore we should not even try to clean up I'm with you on points 1, 2, and 3, but I think point 4 is taking it a bit too far... Surely there must be *some* value in trying to clean up behind yourself, sometimes? Hi I thought it would be good to take this across to python-dev. I've read through https://issues.apache.org/jira/browse/MODPYTHON-109?page=all and the discussion in http://www.modpython.org/pipermail/mod_python/2006-January/019865.html http://www.modpython.org/pipermail/mod_python/2006-January/019866.html and http://www.modpython.org/pipermail/mod_python/2006-January/019870.html again, and I'm just not sure about this. Basically, Apache seems to provide some sort of mechanism for child processes to clean themselves up, and for modules to clean up their resources in a particular child. The argument to remove the ability to clean up Python objects seems to be that: A) The finalize method was been called in an awkward place (from inside a signal handler) and other code may be running and have the GIL, so it may not be called at all, even in a graceful shutdown. B) A normal restart will just send a TERM signal, which doesn't give proper opportunity for cleanup C) If the graceful shutdown doesn't work or respond quickly, Apache will just kill the process anyway, so we may as will live with being killed (talk about mixed metaphors...) D) Since databases etc have to deal with the client process being killed, they generally will handle this I accept that problem A with the finalizing methods is a real problem, but wonder if there are alternate solutions that can be provided to allow cleanups to be attempted. I don't think that B or C is a good argument - in that case why would Apache be providing the hooks to clean up anyway? It feels like throwing in the towel... And D just seems impolite - if we can try and clean up we should. Of course, if we can't manage to call finalize methods even in a graceful shutdown none of this may be possible... Trying to find relevant info on this from the Apache docs and other module documentation: http://httpd.apache.org/docs/2.2/stopping.html
Re: Server Shutdown and register_cleanup
Jim Gallacher wrote: David Fraser wrote: Hi I thought it would be good to take this across to python-dev. I've read through https://issues.apache.org/jira/browse/MODPYTHON-109?page=all and the discussion in http://www.modpython.org/pipermail/mod_python/2006-January/019865.html http://www.modpython.org/pipermail/mod_python/2006-January/019866.html and http://www.modpython.org/pipermail/mod_python/2006-January/019870.html again, and I'm just not sure about this. Basically, Apache seems to provide some sort of mechanism for child processes to clean themselves up, and for modules to clean up their resources in a particular child. The argument to remove the ability to clean up Python objects seems to be that: A) The finalize method was been called in an awkward place (from inside a signal handler) and other code may be running and have the GIL, so it may not be called at all, even in a graceful shutdown. B) A normal restart will just send a TERM signal, which doesn't give proper opportunity for cleanup C) If the graceful shutdown doesn't work or respond quickly, Apache will just kill the process anyway, so we may as will live with being killed (talk about mixed metaphors...) D) Since databases etc have to deal with the client process being killed, they generally will handle this I accept that problem A with the finalizing methods is a real problem, but wonder if there are alternate solutions that can be provided to allow cleanups to be attempted. I don't think that B or C is a good argument - in that case why would Apache be providing the hooks to clean up anyway? It feels like throwing in the towel... And D just seems impolite - if we can try and clean up we should. Of course, if we can't manage to call finalize methods even in a graceful shutdown none of this may be possible... Trying to find relevant info on this from the Apache docs and other module documentation: http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop talks about advising children to exit after their current request. In this case it would seem the cleanup methods should get called at the end of the request processing, and thus shouldn't be in a signal handler (and there should be no other Python code executing...) Except that the parent advises it's children by sending a signal, doesn't it? On Unix it does, but I'm not sure about Win32. Anyway if the exit is not actually not from the signal handler, but the signal handler is simply flagging that an exit should be done after the current request, then the cleanup could be done alongside the exit and outside of the signal handler... http://www.apachetutor.org/dev/pools talks about using pools to allocate/deallocate resources other than memory - could we provide a way to register Python objects that need cleanup using this mechanism? That *is* the mechanism that mod_python uses to register cleanups. req.register_cleanup uses the request pool, and apache.register_cleanup uses the server pool (child_init_pool). Good then :-) Am I barking up the wrong tree or is this worth investigating further? David It's worth investigating. There may be a solution, but we just can't see it. I don't think anyone would argue that the current proposal to drop the server cleanup is sub-optimal, but the current implementation is worse than having no cleanup at all. OK great that's reassuring. I forgot to mention in the above email the mod_perl documentation that seems to indicate that mod_perl does this: http://modperlbook.org/html/ch05_03.html http://162.105.203.19/apache-doc/24.htm#BIN67 http://162.105.203.19/apache-doc/79.htm#BIN172 Really though, isn't this whole discussion actually about database connection pooling? Doesn't that cover 99% of the cases people care about? If so maybe our energies would be better focused on what may be required to support mod_dbd within mod_python. Database connection pooling does seem a large amount of it, but we also do other things from within Apache like launching separate index processes or running things like Excel via COM. At the moment the indexing process watches the parent process and exit when it does, but it might be quite nice to be able to tell the child process it should exit explicitly. http://httpd.apache.org/docs/2.2/mod/mod_dbd.html Although it may be awkward to use mod_dbd with its limited set of database drivers and functionality, when there is the Python DB-API ... I've looked at the mod_dbd documentation before - how do you even execute a SQL statement and retrieve the rows? Maybe I'm missing how it works... But if we could get mod_dbd to manage Python DB-API connections and pool them, now that would be cool as it would require minimal changes to existing Python code... David
[jira] Updated: (MODPYTHON-174) Update requirements to Apache 2.0.47 or greater
[ http://issues.apache.org/jira/browse/MODPYTHON-174?page=all ] David Fraser updated MODPYTHON-174: --- This is now fixed for 3.2.9 (as of rc3) so can be marked fixed... Update requirements to Apache 2.0.47 or greater --- Key: MODPYTHON-174 URL: http://issues.apache.org/jira/browse/MODPYTHON-174 Project: mod_python Type: Task Components: documentation Versions: 3.2.7, 3.2.8, 3.2.x Reporter: David Fraser Attachments: apache-2.0.47-req-doc.patch Usage of the apr_table_compress function requires Apache 2.0.47, but the docs only specify version 2.0.40 See the following URLs for details http://www.modpython.org/pipermail/mod_python/2006-February/020280.html http://www.modpython.org/pipermail/mod_python/2006-May/021135.html http://www.modpython.org/pipermail/mod_python/2006-May/021133.html More details: apr_table_compress was was committed in r263916 to mod_python.c Log message: Fixed crash and memory leak in python_merge_config function in mod_python.c described in MODPYTHON-75. Boyan Boyadjiev provided the code which corrected the problem. This also fixes the memory leak resulting in the use of any PythonOption directive as described in MODPYTHON-60. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MODPYTHON-174) Update requirements to Apache 2.0.47 or greater
Update requirements to Apache 2.0.47 or greater --- Key: MODPYTHON-174 URL: http://issues.apache.org/jira/browse/MODPYTHON-174 Project: mod_python Type: Task Components: documentation Versions: 3.2.7, 3.2.8, 3.2.x Reporter: David Fraser Attachments: apache-2.0.47-req-doc.patch Usage of the apr_table_compress function requires Apache 2.0.47, but the docs only specify version 2.0.40 See the following URLs for details http://www.modpython.org/pipermail/mod_python/2006-February/020280.html http://www.modpython.org/pipermail/mod_python/2006-May/021135.html http://www.modpython.org/pipermail/mod_python/2006-May/021133.html More details: apr_table_compress was was committed in r263916 to mod_python.c Log message: Fixed crash and memory leak in python_merge_config function in mod_python.c described in MODPYTHON-75. Boyan Boyadjiev provided the code which corrected the problem. This also fixes the memory leak resulting in the use of any PythonOption directive as described in MODPYTHON-60. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-174) Update requirements to Apache 2.0.47 or greater
[ http://issues.apache.org/jira/browse/MODPYTHON-174?page=all ] David Fraser updated MODPYTHON-174: --- Attachment: apache-2.0.47-req-doc.patch Added patch to update the docs requirement strings Update requirements to Apache 2.0.47 or greater --- Key: MODPYTHON-174 URL: http://issues.apache.org/jira/browse/MODPYTHON-174 Project: mod_python Type: Task Components: documentation Versions: 3.2.7, 3.2.8, 3.2.x Reporter: David Fraser Attachments: apache-2.0.47-req-doc.patch Usage of the apr_table_compress function requires Apache 2.0.47, but the docs only specify version 2.0.40 See the following URLs for details http://www.modpython.org/pipermail/mod_python/2006-February/020280.html http://www.modpython.org/pipermail/mod_python/2006-May/021135.html http://www.modpython.org/pipermail/mod_python/2006-May/021133.html More details: apr_table_compress was was committed in r263916 to mod_python.c Log message: Fixed crash and memory leak in python_merge_config function in mod_python.c described in MODPYTHON-75. Boyan Boyadjiev provided the code which corrected the problem. This also fixes the memory leak resulting in the use of any PythonOption directive as described in MODPYTHON-60. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: mod_python 3.2.9-rc2 available for testing
I've recently been bitten by the Apache 2.0.47 requirement as mentioned
in the following mails:
http://www.modpython.org/pipermail/mod_python/2006-February/020280.html
http://www.modpython.org/pipermail/mod_python/2006-May/021135.html
http://www.modpython.org/pipermail/mod_python/2006-May/021133.html
I've attached a patch to update the requirement in the docs - would be
good to have this in for 3.2.9
Have also put this in JIRA at
http://issues.apache.org/jira/browser/MODPYTHON-174
Jim Gallacher wrote:
OK, this time for real. :)
The mod_python 3.2.9-rc2 tarball is available for testing. This release
adds support for apache 2.2 as well as some other useful backports from
the development branch. For information on the changes from 3.2.8 take a
look at doc-html/node98.html in the tarball.
Here are the rules:
In order for a file to be officially announced, it has to be tested by
developers on the dev list. Anyone subscribed to this list can (and
should feel obligated to :-) ) test it, and provide feedback *to _this_
list*! (Not the [EMAIL PROTECTED] list, and preferably not me
personally).
The files are (temporarily) available here:
http://people.apache.org/~jgallacher/mod_python/dist/
http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc1.tgz
http://people.apache.org/~jgallacher/mod_python/dist/mod_python-3.2.9-rc1.tgz.md5
Please download it, then do the usual
$ ./configure --with-apxs=/wherever/it/is
$ make
$ (su)
# make install
Then (as non-root user!)
$ cd test
$ python test.py
And see if any tests fail. If they pass, send a +1 to the list, if they
fail, send the details (the versions of OS, Python and Apache, the test
output, and suggestions, if any).
If this tarball looks good, I'll tag it svn and create a 3.2.9 final
tarball.
Thank you for your assistance,
Jim Gallacher
Index: Doc/modpython2.tex
===
--- Doc/modpython2.tex (revision 416697)
+++ Doc/modpython2.tex (working copy)
@@ -16,7 +16,7 @@
\item
Python 2.2.1 or later. Earlier versions of Python will not work.
\item
- Apache 2.0.40 or later (For Apache 1.3.x, use mod_python version 2.7.x).
+ Apache 2.0.47 or later (For Apache 1.3.x, use mod_python version 2.7.x).
\end{itemize}
In order to compile mod_python you will need to have the include files
Re: mod_python 3.2.7 available for testing
Nicolas Lehuen wrote: Hi David, One thing I don't understand is why you have foreign PythonOption definitions in your test setup. Are you sure you have re-created a clean testconf.py file from testconf.py.in ? I've write a little bit of documentation in test/README, please try to follow it. Hi Nicolas I don't understand this either. I followed the docs in test/README (I've checked and did that right). I just repeated the test using my patch to use a different service name (attached). This only generated one segfault and failed on the second cleanup test, but not the first... This behaviour seems repeatable (running the tests twice produced exactly the same results) It seems to me that the test failures all come from this strange test configuration. It looks like mod_python is interfering with some already existing filters, like MultiView. Yes, it does indeed seem that way. So from the point of view, these errors shouldn't hold up the release, as they seem to be problems in the test framework. I just thought I'd like to run them, so it seems a good time to try and solve them. Note that there is a bug under Win32 which I should have reported way before... In some situation, when an exception is raised in mod_python during normal operation, Apache will segfault the next time is is stopped or restarted. It only occurs with a specific exception is raised, maybe when a handler module fails to load. The reason why I haven't reported it yet is that it only occured on my development computer, not my production servers, so I thought it was something caused by my setup. Apparently you have the same problem, hence the segfault when the Apache server is stopped. I'll have to investigate this problem more thoroughly. Interesting that it seems to make a difference running under a different service name. I can try help track this down. So please try to re-run the test in a clean slate environment. It'd be interesting to see why and how your current configuration contaminates the test configuration, though. OK I'm busy setting up VMWare to try this - pretty sure it'll pass, so I'll try investigate whats getting the situation confused... Thanks for your feedback David Index: test/testconf.py.in === --- test/testconf.py.in (revision 375881) +++ test/testconf.py.in (working copy) @@ -18,6 +18,9 @@ # Path to the mod_python.so file, should be in the apache modules directory. MOD_PYTHON_SO=@MOD_PYTHON_SO@ +# Set SVCNAME if you want to test via the given service +SVCNAME= + # makes emacs go into python mode ### Local Variables: ### mode:python Index: test/test.py === --- test/test.py (revision 375881) +++ test/test.py (working copy) @@ -176,6 +176,7 @@ TESTHOME = testconf.TESTHOME MOD_PYTHON_SO = testconf.MOD_PYTHON_SO LIBEXECDIR = testconf.LIBEXECDIR +SVCNAME = testconf.SVCNAME SERVER_ROOT = TESTHOME CONFIG = os.path.join(TESTHOME, conf, test.conf) DOCUMENT_ROOT = os.path.join(TESTHOME, htdocs) @@ -299,12 +300,38 @@ f.write(\n# --APPENDED-- \n\n+append) f.close() +def installService(self,extra=''): + +print Installing Test Apache Service +httpd = quoteIfSpace(HTTPD) +config = quoteIfSpace(CONFIG) +cmd = '%s %s -k install -n %s -f %s' % (httpd, extra, SVCNAME, config) +print , cmd +os.system(cmd) +time.sleep(1) +self.httpd_installed = 1 + +def uninstallService(self,extra=''): + +print Uninstalling Test Apache Service +httpd = quoteIfSpace(HTTPD) +cmd = '%s -k uninstall -n %s' % (httpd, SVCNAME) +print , cmd +os.system(cmd) +time.sleep(1) +self.httpd_installed = 0 + def startHttpd(self,extra=''): +if SVCNAME: +self.installService(extra) print Starting Apache httpd = quoteIfSpace(HTTPD) config = quoteIfSpace(CONFIG) -cmd = '%s %s -k start -f %s' % (httpd, extra, config) + if SVCNAME: +cmd = '%s %s -k start -n %s' % (httpd, extra, SVCNAME) +else: +cmd = '%s %s -k start -f %s' % (httpd, extra, config) print , cmd os.system(cmd) time.sleep(1) @@ -315,7 +342,10 @@ print Stopping Apache... httpd = quoteIfSpace(HTTPD) config = quoteIfSpace(CONFIG) -cmd = '%s -k stop -f %s' % (httpd, config) + if SVCNAME: +cmd = '%s -k stop -n %s' % (httpd, SVCNAME) +else: +cmd = '%s -k stop -f %s' % (httpd, config) print , cmd os.system(cmd) time.sleep(1) @@ -337,6 +367,9 @@ self.httpd_running = 0 +if SVCNAME: +self.uninstallService() + class PerRequestTestCase(unittest.TestCase): appendConfig = \nNameVirtualHost *\n\n @@ -1872,6 +1905,8 @@ def
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
Jim Gallacher wrote: Barry Pederson wrote: I think this is the general kind of thing we're looking for though, with some mistaken pointer/memory operation. Too bad we can't write *everything* in python. :( You haven't been following PyPy then? :-) David
Re: Testing mod_python on win32
Hi Nicolas Of course, one way of fixing this up is ensuring we use a test framework that cleans everything up. As it stands, we could potentially have Apache processes left hanging around... In fact we could have a service left running that seems to be the standard Apache service but is in fact the test service... I think the current situation is worse since it actually stops your existing Apache service if you leave it running. At least with the patch the stuff left hanging around is clearly marked as testing mod python. The alternative would be to test Apache standalone rather than as a service, but this would make the tests more difficult to control David Nicolas Lehuen wrote: David, Though your code seems perfect, I'm a bit worried about installing a service, even temporary, for testing purposes. It adds another point where the test could fail for setup reasons, and should the test end unexpectedly, the tester's system has an extra service which points to a temporary generated configuration file. Since everything seems OK without the service layer, I'd rather we keep the test suite as is and not introduce some extra feature that we'll have to debug later on. Nothings kills me more than debugging the tests :). Regards, Nicolas 2005/12/6, David Fraser [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Hi Nicolas Interesting, I never even tried running it the same time as the service, so I can see that actually works now (except for the monitor problem). Yes, my patch fixes the problem with the monitor David Nicolas Lehuen wrote: Woops, I've marked the issue as resolved since I've corrected the documentation, but I've not taken your patch into account. I'm not really sure it's required, though, since the only problem is that the Apache Monitor gets a bit confused by the test server, buit everything else works correctly. Does your patch fixes the problem with the Apache Monitor ? Regards, Nicolas 2005/12/6, David Fraser [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: I'm sure this is Win32 only. We could even remove the requirement on Win32 by using an alternative service name that we create and destroy as required. I've opened a Jira issue with a patch for this: http://issues.apache.org/jira/browse/MODPYTHON-95 Graham Dumpleton wrote: I'm a bit confused by: - The only trick is that you'll have to stop your Apache server before launching the test, as the start/stop command can only apply to one single Apache instance. Does this apply to UNIX as well as Win32? I ask as I have never bothered to explicitly shut down any running instance of Apache, yet haven't noticed any problems with running the tests. If this is a Win32 specific instruction, you might want to note it as such. On UNIX systems, where the web server may be doing real work, people may not want to shut it down just to be able to test a new separate version of mod_python that hasn't been installed yet. Graham On 06/12/2005, at 8:02 AM, Nicolas Lehuen wrote: Hi David, To follow my old promise, I've just checked in a bit of documentation on how to run the test suite, including on Win32. I've also added a few self-test in the test module, so that the most obvious setup mistakes are notified to the user. Here is the documentation, directly from the Subversion repository : http://svn.apache.org/repos/asf/httpd/mod_python/trunk/test/README This should eventually be converted to TeX and integrated into the real documentation, but for various reasons this way is the quickest way to put it online. It's much better than the previous README file anyway (it was basically saying keep out unless you know what you're doing ;). Hope this helps. Regards, Nicolas 2005/12/5, David Fraser [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: As afar as I can recall, Nicolas Lehuen is the only guy who's been able to run the tests on win32 Has anybody else been able to? Can we put together some hints as to how to do it? David
Testing mod_python on win32
There's an old thread below about testing on Win32. As afar as I can recall, Nicolas Lehuen is the only guy who's been able to run the tests on win32 Has anybody else been able to? Can we put together some hints as to how to do it? David David Fraser wrote: Nicolas Lehuen wrote: Being the guy who provide the Win32 binaries, I feel obliged to answer :) Brilliant :-) download, untar, then do (substituting your apache directory for the one below) cd dist set APACHESRC=c:\Program Files\Apache Group\Apache2 build_installer.bat Note that without setting APACHESRC, the setup will try and locate apxs to find the include and library directories, which will fail on a default (non-source) apache install (and which requires running configure which is wierd on windows anyway...) Perhaps a note on the could be added to the README? Yes, and I'll add a test to build_installer.bat which will display a nice error message when APACHESRC is not set. Great, thanks Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Again, you can't do this on Windows without running configure. Did those who tested on Windows run configure or how did they get it to work? I'm curious as to the setup :-) No need to run configure (which would cause a bunch of problems on Windows). Just copy testconf.py.in to testconf.py and replace the @..@ macros manually. Surely we could write code to figure these out? We work out many of them in setup.py.in / win32_postinstall.py anyway... I tried building myself and testing the py2.4 installer, but with both the tests failed to start the Apache service and so universally failed. This was with a manually created testconf.py I'm not giving a -1 until I know I'm doing the right thing though :-) You're doing the right thing if you have something like this in testconf.py : HTTPD=r'c:\apache\bin\apache.exe' TESTHOME=r'D:\projets\mod_python\test' MOD_PYTHON_SO=r'C:\apache\MODULES\mod_python.so' LIBEXECDIR=r'C:\apache\modules' In any case this would be a -1 on the test framework, not on mod_python... OK - it seems the test framework leads at least a bit of documentation as to how to run on Windows And I'd like to provide a way of doing this on Windows without configure if possible All right, all right, I'll add a few lines of documentation to test/README. Nothing like bothering people :-) So do these tests run for you? The disturbing thing for me was that the seem to fail without any error messages being produced by apache, even in the test/logs/* files It turns out that if the service can't write to the error log file, it fails and logs a message in the Windows Event log, rather than to the console. This is odd because it actually opens the log files itself. I suspect this is because it actually runs using a different user as a service... Anyway I should have been testing the earlier betas so I'm doing catchup :-) Cheers David
Re: Testing mod_python on win32
Hi Nicolas Interesting, I never even tried running it the same time as the service, so I can see that actually works now (except for the monitor problem). Yes, my patch fixes the problem with the monitor David Nicolas Lehuen wrote: Woops, I've marked the issue as resolved since I've corrected the documentation, but I've not taken your patch into account. I'm not really sure it's required, though, since the only problem is that the Apache Monitor gets a bit confused by the test server, buit everything else works correctly. Does your patch fixes the problem with the Apache Monitor ? Regards, Nicolas 2005/12/6, David Fraser [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: I'm sure this is Win32 only. We could even remove the requirement on Win32 by using an alternative service name that we create and destroy as required. I've opened a Jira issue with a patch for this: http://issues.apache.org/jira/browse/MODPYTHON-95 Graham Dumpleton wrote: I'm a bit confused by: - The only trick is that you'll have to stop your Apache server before launching the test, as the start/stop command can only apply to one single Apache instance. Does this apply to UNIX as well as Win32? I ask as I have never bothered to explicitly shut down any running instance of Apache, yet haven't noticed any problems with running the tests. If this is a Win32 specific instruction, you might want to note it as such. On UNIX systems, where the web server may be doing real work, people may not want to shut it down just to be able to test a new separate version of mod_python that hasn't been installed yet. Graham On 06/12/2005, at 8:02 AM, Nicolas Lehuen wrote: Hi David, To follow my old promise, I've just checked in a bit of documentation on how to run the test suite, including on Win32. I've also added a few self-test in the test module, so that the most obvious setup mistakes are notified to the user. Here is the documentation, directly from the Subversion repository : http://svn.apache.org/repos/asf/httpd/mod_python/trunk/test/README This should eventually be converted to TeX and integrated into the real documentation, but for various reasons this way is the quickest way to put it online. It's much better than the previous README file anyway (it was basically saying keep out unless you know what you're doing ;). Hope this helps. Regards, Nicolas 2005/12/5, David Fraser [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: As afar as I can recall, Nicolas Lehuen is the only guy who's been able to run the tests on win32 Has anybody else been able to? Can we put together some hints as to how to do it? David
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
Gregory (Grisha) Trubetskoy wrote:
Having looked at the FieldStorage code, I'm guessing the idea was that
you parse fields as they come in and append them to a list. This
preserves the original order of fields, in case it is needed.
I'm not sure that maintaining a dictionary alongside the list is the
right thing to do. It might be, but there are some difficult questions
to answer -e.g. how costly is a sequential search, and is the code
complexity (and fieldstorage code is no picnic to read as it is) worth
the speedup?
Also while it would speed up retrieval, it will slow down the write
operation - when a field is added to fieldstorage you now need to
append it to the list, AND check whether it exists in the dictionary,
then add it there as well.
How often do developers access form fields via __getitem__? I noticed
the publisher does not use it - it iterates the list, so nothing would
be gained there.
We do it a lot but we copy it into a different dictionary first to get
exactly the setup we want. But dictionary-style access is a very
obvious, pythonic way to do it.
I have a simple 70-line ordereddict implementation which is derived from
dict and remembers the keys in the order that they were assigned when
iterating through the list, this may be a way to go for this. It just
uses a list of keys internally to remember the order, and otherwise is a
dictionary...
Also, something else to consider - is there a simple programatic
solution that could be documented, e.g. something like
my_fs = util.FieldStorage(req)
dict_fs = {}
dict_fs.update(my_fs)
[have no idea whether this will work :-)]
It may work but still has the potential performance problem since it
loops through the keys and then does a getitem on each key which loops
through them again. Not likely to cause problems for a small number of
arguments but not ideal :-)
and voila - you've got a dictionary based fieldstorage?
Anyway, just a few cents from me.
Grisha
[jira] Commented: (MODPYTHON-94) Calling APR optional functions provided by mod_ssl
[ http://issues.apache.org/jira/browse/MODPYTHON-94?page=comments#action_12358754 ] David Fraser commented on MODPYTHON-94: --- I wonder whether it would be possible to use a module like ctypes to connect to Apache functions. That way the linking issue is not a problem... Calling APR optional functions provided by mod_ssl -- Key: MODPYTHON-94 URL: http://issues.apache.org/jira/browse/MODPYTHON-94 Project: mod_python Type: New Feature Components: core Versions: 3.2 Environment: Apache 2 Reporter: Deron Meranda Attachments: modpython4.tex.patch, requestobject.c.patch mod_python is not able to invoke APR Optional Functions. There are some cases however where this could be of great benifit. For example, consider writing an authentication or authorization handler which needs to determine SSL properties (even if to just answer the simple question: is the connection SSL encrypted). The normal way of looking in the subprocess_env for SSL_* variables does not work in those early handler phases because those variables are not set until the fixup phase. The mod_ssl module though does provide both a ssl_is_https() and ssl_var_lookup() optional functions which can be used in earlier phases. For example look at how mod_rewrite calls those; using the APR_DECLARE_OPTIONAL_FN and APR_RETRIEVE_OPTIONAL_FN macros. I can see how it might be very hard to support optional functions in general because of the C type linkage issue, but perhaps a select few could be coded directly into mod_python. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Developer Guidelines (was Re: More efficient FieldStorage, StringField and Field classes)
Mike Looijmans wrote: David Fraser wrote: See my other mail - basically you do cd dist set APACHESRC=c:\path\to\apache (where this is the Apache2 dir in a standard installation, now need for apache source) build_installer Okay, first I downloaded the .NET SDK. (Our internet connection is great here) Then I get: error: Python was built with version 7.1 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed. I don't have a VS license - that would mean I have to build my own Python in order to get any further. Oh dear... Strange, I have VS.NET 2003 installed here, and it doesn't give errors. I think I got mod_python compiling at one stage with mingw so that is the other route you could go... I was wondering - since i won't be hackin' the C code (yet), would it be sufficient to just install the binary, and use the .py files as obtained from the SVN repository? Yes you should be able to install the binary and run the tests from the repository (if you can get past the issues I had running them ...) The 3.2.5 beta seems to run just fine here. Cannot run the automated tests though, because of the issues mentioned above. Cheers David
Re: Solving the import problem
Graham Dumpleton wrote: On 10/06/2005, at 2:53 AM, dharana wrote: As for vampire - why would I want vampire? mod_python is great except this. I personally have no interest in adding yet more software to my system just to solve the mod_python import issue - Id rather it was fixed in the right place...not everyone uses vampire... I really get annoyed by so much Vampire ads everywhere. I understand it's developers have spent considerable time in it but I think people who come to python for webdev from an easier framework do it because they want _more_ control, not less (at least that is my case). I am happy with my custom framework in python now, it didn't took me a lot of time and it's tuned for my needs. I won't look into using Vampire for that reason. Vampire is not about giving you less control, it is actually the opposite. It gives you more glue components and hooks so as to give you more control and more and better ways of doing things over what mod_python by itself provides. It is not just some monolithic blob and isn't intended to be a framework where you are restricted to working in a certain way. I'll quit with the advocacy if that is what people want, but it gets pretty disheartening when you see people on the mailing list trying to solve problems, and not really understanding properly how to do it, when Vampire already provides an example of how to do it or a pre-canned solution, yet you can't even get them too look at it. I have continually found that it is like the saying you can lead a horse to water, but you can't make it drink. :-) I'll shut up for a while now. I think it's great you've solved some of these problems in Vampire... Some of the solutions should definitely be brought back into mod_python, using Vampire as a staging ground for mod_python improvements as you recommended earlier. David
Re: [jira] Commented: (MODPYTHON-55) Add a version attribute to the mod_python module.
Hmmm ... it appears I was over-optimistic about my patches :-) Most of them related to win32 installation rather than building. There are a few changes to setup.py.in that rearrange it but most are to the win32_postinstall.py I had a power failure two days in a row so sorry for the delay ; please carry on! David Fraser wrote: I've got a number of patches which relate to building on Windows using pure python. I'll try get them onto Jira next week David Nicolas Lehuen wrote: The problem is that this won't work when building on Windows. I think we should use Python scripts instead of configure, grep et al which are not truly portable (well, I hear you, portable everywhere except on Windows :)). Regards, Nicolas 2005/5/20, Jim Gallacher (JIRA) [EMAIL PROTECTED]: [ http://issues.apache.org/jira/browse/MODPYTHON-55?page=comments#action_65861 ] Jim Gallacher commented on MODPYTHON-55: Would it be appropriate to determine the version at configure time? If so we could create a file mod_python/__init__.py.in which would contain: version=@MP_VERSION@ and configure would grep src/include/mpversion.h for MPV_STRING and generate __init__.py from __init__.py.in. If this seems reasonable I can put something together. Add a version attribute to the mod_python module. - Key: MODPYTHON-55 URL: http://issues.apache.org/jira/browse/MODPYTHON-55 Project: mod_python Type: Improvement Versions: 3.1.4 Reporter: Nicolas Lehuen Assignee: Nicolas Lehuen Fix For: 3.2.0 (asked by dharana) There is no easy way to know the version of mod_python used from within an handler. Why not try to add a version string attribute to the mod_python module ? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
