Re: Constructing of a URL for location redirect.
Nicolas Lehuen wrote: > BTW, did we ever considered using SWIG to map the Apache API ? I know > it can be quite tricky to use, but it could be a real time saver. That's essentially what mod_snake did, and why I liked it so much. Though I don't remember if it used swig or pyrex. Nick
Re: Constructing of a URL for location redirect.
Gregory (Grisha) Trubetskoy wrote: > SWIG in my opinion is good when you want some kind of an API made > available to you quickly, but in a static environment where can put some > thought into functionality, usability, Pythonic-ness of every approach, > write documentation with good examples and a meaningful test case SWIG > does not buy much. > > If someone just wanted to map the APR to Python - they can always use > SWIG, but that's not what mod_python is about. All true; mod_python is a layer up from the apache API, so there's no way the SWIGed API could come close to replacing it. I just found it generally more convenient to work in Python rather than C, even if it's only a thin layer above that, and I think there is some merit to using it in some places, carefully wrapped in a Python class or module. That said, I don't really see mod_python changing so radically at this point in the game. It's fun to think about, though. :) In a masochistic kind of way. Nick
Re: Vote on whether to integrate server side include (SSI) support.
+0 Graham Dumpleton wrote: > Anyway, the point of this email is to get a decision on whether this > major new feature should or should not be added into mod_python. > > Core developer votes obviously matter the most, but others are more > than welcome to voice an opinion.
Re: FieldStorage and multiline headers in multipart/form.
Mike Looijmans wrote: > Just a thought: Can we (re)use the rfc822 mime parser that's already > built-in in Python to do the work for us? I believe the standard cgi module does just that. I'm probably going to hell for having been importing cgi and replacing parse_qs[l] with the versions from _apache. Nick
Re: TLP Name
Gregory (Grisha) Trubetskoy wrote: > PythonScript > Pythonidae > PyPache > pythonalia > Quetzalcoatl > Asphyxia > Scales > Pythonistas > PigeonPy > Pungi I liked Pungi, but there is unfortunately another project out there with that name (https://hosted.fedoraproject.org/projects/pungi). Quetzalcoatl is a great Apache project name in that it's kind of related in a cultural memory kind of way to the project ideas, but obscure enough to be completely forgettable :) I guess Scales also falls into that category, and it gets my vote because it's easier to spell. Nick
Re: TLP Name
+1 Quetzalcoatl
Re: Solving the import problem
On Thu, 2005-06-09 at 00:50 +0200, Nicolas Lehuen wrote: > Most of the problems raised when importing handlers (including double > import of handlers) have been solved in the trunk for a while now, the > bug fixes are waiting for an official release. Looks good, seems to work well. However, I never use import_module myself in any of my code, so I'm never going to interact with the dict that holds the handler modules. > But like I've wrote > before, the current implementation of import_module is still broken > and dangerous for our users. I really don't see how you're going to get around using an import hook for that, at least temporarily while import_module is called so all the dependent imports get reloaded, unless it's a "standard" Python library (i.e. /usr/lib/pythonX.X or wherever Python is installed). I also don't think there's anything wrong with operating directly on the sys.modules dict if you're properly using the global import lock. > It's time to change it before it's too late (before compatibility > issues force us to leave it as is). I'll try to spend some time on it > on next sunday. I think the syntax is OK, just the implementation that needs changing. I don't think there should be any compatibility issues. Nick
Re: Solving the import problem
Graham Dumpleton wrote: Note that the direction I am looking at here is that apache.import_module() is made to function properly in the contexts it needs to and not perform double duty in satisfying extra requirements of top level mod_python imports where it has to import stuff from site-packages. The top level imports should be treated specially and it should only defer to apache.import_module() for imports from the document tree. Yes, I was thinking exactly along these lines myself. I have never really looked that closely at the import_module code since I never used it, but I felt I might have something useful to contribute if I did. After looking closely at it, I think you're right. The handler module loading should be handled separately and maintained separately. "Autoreloading" a handler module should be as simple as just deleting the module and everything that came with it, and then calling the loader again. If this separation is done, I think that the distinction that has been introduced with a separate module loader in mod_python.publisher can be eliminated. The apache.import_module() can simply be replaced with that in mod_python.publisher or a modification of it to satisfy other requirements I will talk about later in future emails. I agree; the code there is relatively simple and will probably do the job for most people. Keep it simple, and it can serve as an example for people who want to do something more sophisticated. Or they can upgrade from publisher to vampire; honestly, if you like publisher I don't know why you *wouldn't* want to upgrade to vampire :) As far as imports from any of the above imported modules goes, the general rule should be that if it is a standard module in sys.path, then "import" is used. If it is within the document tree then apache.import_module(). Right, that's what I was trying to get at in my earlier email. Just avoid the problems of messing with the standard libraries altogether. As far as utility modules which exist outside of the document tree which are specifically related to the web application but which aren't on sys.path and for which you want module reloading to work, apache.import_module() would still be used, but you have to specify the actual directory to the function. In some respects the ability not to specify a path to apache.import_module() should be disallowed with a path always required. Further, sys.path should no longer be automatically ammended to include the directory where the PythonHandler is defined for. And apache.import_module() should never search in sys.path. Exactly... apache.import_module should be used only to load support modules for your application, not in general for importing. What you're doing with apache.import_module can have unforeseen side effects that you don't expect, so keeping its use restricted to ONLY modules that you KNOW need to be reloaded is the best policy. And requiring the path argument will go a long way in enforcing this. The implication of not extending sys.path automatically is that "import" will not work to load a file in the same directory as the handler when in the document tree. This was always dangerous anyway as that module could also have been loaded by apache.import_module() and a problem could thus arise. If "import" is used in this way it would need to be changed to apache.import_module(), or a simple import hook introduced which when used in a module imported using apache.import_module() will use apache.import_module() underneath for an "import" of a file in the same directory. I think that's a perfectly acceptable trade off, and it avoids potential problems that exist with the current code. I still don't necessarily think and import hook is necessary, as you've got to follow *some* conventions when you're working within a framework. And, we're really only talking about people who are going to use the handlers provided with mod_python. How does this seem to people? There is stil more detail just in this bit which will need clarification and there are other issues as well which I haven't even mentioned. This all looks good to me. I hate to just say "yes I agree" to everything without really adding much to the discussion, but you've clearly been thinking about it a lot longer than most people. Nick
Re: Solving the import problem
Graham Dumpleton wrote: Handler modules in your document tree is done all the time, you can't avoid it with mod_python.publisher and most other systems. I think you misunderstand why I'm saying, or maybe I misuderstood Nicolas. By "Handler" modules I mean your modules that implement accesshandler, authenhandler, handler, etc. mod_python.publisher is one such module. vampire is another. At least that's what I understand you to be importing in the context of this statment: > I've understood you point, but there is a difficulty in judging from a > PythonHandler directive whether the handler should be loaded as a > standard Python module, from the sys.path, or as a dynamic Python > module, from the document tree. Nick
Re: Solving the import problem
Graham Dumpleton wrote: module, from the document tree. Maybe the context of the directive could be used for that ; if the directive is defined at the server or virtual host level, then it's a top level handler, otherwise if it is defined in a Location or Directory (or .htaccess file), then it's a handler that should be loaded from the document tree (with a possible fallback to sys.path if it is not found ?). Aside from the potential security issues of storing your handler modules in the document tree, I just don't it's a good idea to traverse the document tree for a python module/package. Even in a shared hosting situation, there are still ways to store your modules outside the accessible document tree. I can just see so many confused people wondering why module B was imported instead of module A, which reside in different parts of the document tree. And module B isn't even a handler, it's a support module. Ugly. Not to mention name collision problems that are bound to happen when per directory interpreters aren't being used in that situation. The big problem now is that if the __init__.py file uses standard import statement with the expectation that it will grab the module or package from within the same directory, it will not work. This is because to the Python import system it will not know that it is to be treated as a package and look in that local directory first. Among other minor "gotchas" that crop up from time to time, but you've hit the big one. mod_python itelf isn't a framework as such; I'm for making mod_python accessible and usable and all that, but isn't it fair to say that mod_python doesn't have to solve *all* import problems? Handling imports automagically from your imported modules requires an import hook... I got past this problem in Vampire through the use of the import hook. Vampire would stash a special global object in the module so the import hook knew that it was a special Vampire managed module and would grab the module from the local directory and import it using the Vampire module importing system rather than standard Python module importer. At the moment though this only works at global scope and not when import is used in the handler code when executed, although can probably solve that. We can probably do some checks to see if we're importing a single module or an entire package without resorting to import hooks. Deal with the modules in __all__ (from __init__.py) like we'd handle single module imports, and any explicit "imports" in the package beyond that are not reloadable. Otherwise they can explicity call apache.import_module. Document this behaviour and keep it simple, otherwise people are not going to be able to debug their problems easily. Not to mention that the code will grow into a beast. Although from/import syntax also works, if it tried to import "a.b" from a subpackage, it will not work if "b" wasn't explicitly imported by "a" to begin with. I haven't generally experience that with import hooks, although os (and therefore os.path) seems to peskily not import because of some sys.modules manipulation weirdness. The question thus is, if you understand what I am raving about, is whether it is reasonable that packages will not be supported by apache.import_module(). There is a slim chance some ones code may break as a result but the majority would work fine. See above. Nick
Re: PythonSessionOption - a new apache directive for session configuration
How about an explicit "None" value to completely disable it? If you don't
want users on your site using it.
Nick
Jim Gallacher wrote:
So, any further thoughts / comments / objections to PythonSessionOption,
or shall I just check in the code?
Regards
Jim
Jim Gallacher wrote:
I've created a new apache directive called PythonSessionOption. This
would be used to configure session handling in the apache config file.
This data is accessed with a new request method,
req.get_session_options().
Although we could use the PythonOption directive instead of creating a
new one, I believe it's better to keep the session config data
separate so we don't need to worry about collisions with current user
code or configuration.
Typical Usage
-
In a test script mptest.py
def handler(req)
opts = req.get_session_options()
for k in sess_conf:
req.write('%s: %s' % (k,opts[k])
In Session.FileSession:
__init__(self,req,sid):
opts = req.get_session_options()
timeout = int(opts.get('timeout', DFT_TIMEOUT))
In an Apache config file:
ServerAdmin [EMAIL PROTECTED]
ServerName example.com
DocumentRoot /var/www/
PythonSessionOption session FileSession
PythonSessionOption session_directory /var/lib/mod_python/sess
PythonSessionOption timeout 14400
PythonSessionOption lock 1
...
If there are no objections I'll commit the code. I have not refactored
Sessions.py to use the new configuration scheme just yet.
Regards,
Jim
Re: PythonSessionOption - a new apache directive for session configuration
Jim Gallacher wrote: Nick wrote: How about an explicit "None" value to completely disable it? If you don't want users on your site using it. Do you mean to disable sessions, or just the session configuration? Yes, I'm sorry, I mean disable the session mechanism altogether. Some admins may see it as a potential security issue. Although I'm sure you're doing the best you can to make sure it can't be exploited as such, if you make it so people can evenutally plug in their own session mechanisms, there might be room for abuse. Nick
Re: PythonSessionOption - a new apache directive for session configuration
Jim Gallacher wrote: Just so I'm *really* clear, do you mean the current scheme for session handling would also be disabled? The more I think about it, you're right; you can just set up the session stuff without directives just the same by importing mod_python.Session and going from there. So that line of reasoning has no merit. I was confused by the adding of a new directive, which seems to indicate that there would be some "default" Session handler being loaded if you didn't specify one using the directive. Because in the old scheme there wasn't really any "default" session handling, unless you outright imported the libary and started using it. If it's going to work exactly the same way as it did before, except now you can configure some defaults in the apache config, then I'm probably worried over nothing. But it probably should have an option for "None" or "disabled," which would be the default, meaning I don't care to use the supplied session handlers, even though it didn't really do anything extra than it does now. But in that case, why not use "PythonOption session_ ", which is probably what you were asking about in the first place, which I think someone else mentioned as well. That doesn't imply that there is some kind of default session handling, just the standard way of passing values from the apache config to python code. If the plan is to implement a pure C session handler, then PythonSessionOption makes sense, but otherwise it doesn't seem necessary. Nick
Re: PythonSessionOption - a new apache directive for session configuration
Nicolas Lehuen wrote: +1 for PythonOption session_ Unless choosing a specificc configuration directive has something to do with security (i.e. no overloading of the settings in .htaccess files) ? (This is what I was ineptly trying to get at earlier.) Nick
Re: Session Benchmarks
Right, I was thinking the same thing... those are pretty extreme conditions. But very interesting as a benchmark. I assume ext3? Nick dharana wrote: Wow, good work Jim. A bit of perspective: 5 active sessions with an average of 15 mins duration per session means roughly 4,800,000 requests a day or 55 requests per second. Jim Gallacher wrote: Hello All, In testing the new req.get_session() method for the upcoming 3.2.0 release I've noticed that the performance of FileSession degrades badly as number of session files rises. Assuming this is related to putting a large number of files in a single directory, I decided to do some benchmarking outside of mod_python to investigate further. I tested 4 different persistent stores on a linux system (kernel 2.6.9, ext3 file system). A simple dict was used to represent a typical session object. The session id is generated using the same method found in mod_python/Session.py, but adapted for use outside of mod_python. Tests = Dbm --- Pickle saved in a dbhash (Berkley DB) table. The db table is opened and closed for each record saved. This is the same as the current implementation of DbmSession. mysql - Pickle saved as a blob in a mysql table. A new connection is opened and closed for each record saved. mysql2 -- Same as mysql but only one connection is opened for the complete test. Gives an indication of the overhead resulting from opening/closing 1000 connections. FS -- Pickle saved as a file in a single directory. This is the same as the current implementation of FileSession. FS2 no_sync --- Pickle saved as a file in a directory derived from the session id. There are 256 possible subdirectories. eg. sess/00/00fe9c4b32bcb01b60a61cc674aa0ac9 sess/ab/abff1785c78200baaa59e683da4038dd FS2 sync Same as FS2 no_sync, but /bin/sync was called after 1000 files were created to flush the OS write buffer. Discussion == For comparison, apache on the system used for these tests can serve plain text files at 1650 req/second, or mod_python req.send_file() requests at 850 req/second. Reading and writing the session object is a possible limiting factor. By examining the results below we can estimate the upper bound that may be imposed when session handling is enabled and the impact of a give session storage mechanism. FS -- The FS store shows O(n) performance. For a small number of session files the performace is good. For the first 1000 session files our upper bound would be 2857 requests/second. There will be little impact on performance since this is faster than apache can serve a page. At 50,000 files the *best* you could expect within mod_python is 72 requests / per second. This will obviously have a severe impact on performance. The current FileSession implementation scales poorly and needs to be re-written. FS2 --- Using the multiple directory layout gives a significant performance improvement over a single directory as the number of session files increases. For 5 session files it looks like O(1) behaviour. (This is not strictly true as the behaviour is really O(n), but this does not become significant until the number of session files rises beyond 5x10**5). Using this storage scheme imposes an upper bound of 2000 requests/second with the following caveat. The write buffers of the underlying OS need to be flushed to disk. The time variations seen in the table (FS2 run 1, run 2) are a result of this syncronization at random intervals, which imparts a significant performance penalty. See row 47000 for a really agregious example - 22 seconds for 1000 files. The FS2-sync test shows that the storage scheme scales well for a large number of files, but since we can't control file syncing in a production environment there will be a performance penalty at high request levels. FileSession with the FS2 directory layout will give the fastest response time, but only when the server is under a light to medium load. What constitutes a medium load will depend on the underlying OS and it's IO subsystem. Dbm --- The dbhash table shows O(n) behaviour and does not scale well for a larger number of session files, although it is better than the current FileSession. The benchmarks used in this study did not do any file locking, but in the actual DbmSession implementation only one process/thread is allowed access to the dbm file at a time which will have a negative impact on performance. Likewise, there are no obvious optimizations for removing expired sessions, which is another potential bottleneck. The best case for Dbm is 606 req/second at the 1000 file mark, and 122 req/second at the 50,000 file mark. MySQL - The data for a mysql backend suggests O(1) behaviour. It may not be obvious from the table, but there is a cost when the mysql db flushes it's records to disk. For the number of records inserted in this test the time for
Re: Session Benchmarks
Gregory (Grisha) Trubetskoy wrote: I'm a little concerned about staying focused on closing the last bugs so that we get to a point where a release can be made, since there hasn't been one in such a long time... +1 on that Nick
Re: Session Benchmarks
Jim Gallacher wrote: Using bsddb3 would introduce new dependency for mod_python, so I don't know if it's a good idea to use transaction handling by default for DbmSession. Maybe we could offer a subclass? Starting with Python 2.3 this module is included in the standard python distribution as its bsddb module. Nick
Re: Session Benchmarks
Nick wrote: Jim Gallacher wrote: Using bsddb3 would introduce new dependency for mod_python, so I don't know if it's a good idea to use transaction handling by default for DbmSession. Maybe we could offer a subclass? Starting with Python 2.3 this module is included in the standard python distribution as its bsddb module. I just upgraded my session handling routines to make use of a bsddb.dbobj.DBEnv object, and it works like a champ. Does mod_python need to support Python 2.2 anymore? Perhaps there is already new code that precludes 2.2, I haven't looked. Nick
Re: Session Benchmarks
Jim Gallacher wrote: Nick wrote: Jim Gallacher wrote: Using bsddb3 would introduce new dependency for mod_python, so I don't know if it's a good idea to use transaction handling by default for DbmSession. Maybe we could offer a subclass? Starting with Python 2.3 this module is included in the standard python distribution as its bsddb module. Are you sure? The docs on python.org indicate otherwise. It would be sweet if bsddb3 was the std module. Jim Directly from the Python module documentation: "Starting with Python 2.3 the bsddb module requires the Berkeley DB library version 3.2 or later (it is known to work with 3.2 through 4.3 at the time of this writing)." Nidk
Re: flex [was mod_python 3.2.0-BETA available for testing]
Jim, I don't think it's too verbose, but maybe you could delay it to the end of the configure script so you don't have to either interrupt with control-C or scroll back to see what went "wrong." Here's another idea: Fail the flex test fairly silently (e.g. just "no"), but fall back to a script that generates a nice, verbose error message explaining the situation. That way, when the user tries to call "make" after modifying the .l file, the fake flex alternative script gets called, displays the message, and exits with status 1. Nick Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: OK, here is the flex scoop - as the the docs point out, anything before 2.5.31 is not reentrant and I think even uses a slightly different interface so older flex won't even process the psp_parser.l file correctly. Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 > 2.5.4 because 31 > 4 - I for a while had trouble seeing that for some reason), so the new flex is still not commonplace. So until reentrant flex becomes commonplace, the solution was to include a pre-parsed psp_parser.c so that you woudln't need flex at all to compile mod_python. Looks like this still should be the case. The ./configure should just print a warning that if flex is not found or too old, should you need to rebuild psp_parser.c you will need to get the right version of flex. I've made the changes to configure.in, but before I commit I wanted to get some feedback. Are the following configure messages unclear or too verbose? For the case of the missing flex, ./configure will generate: ... checking for --with-flex... no checking for flex... no configure: WARNING: flex not found You can generally ignore this warning unless you need to regenerate psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c, use --with-flex to specify the location of flex. See the README for more information. ... For the case of the wrong flex version, ./configure will generate: ... checking for --with-flex... no checking for flex... /usr/local/sbin/flex found /usr/local/sbin/flex, we'll use this. Use --with-flex to specify another. checking flex version... configure: WARNING: Flex version 2.5.4 found. Version 2.5.31 or greater is required. You can generally ignore this warning unless you need to regenerate psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c, use --with-flex to specify the location of the correct flex version. See the README for more information. ... Any comments? Jim
Re: SQLite and other things [was Re: svn commit: r290569]
Gregory (Grisha) Trubetskoy wrote: I think that (and we can discuss this - I don't set laws, I just have opinions that may not always beright, so feel free to comment) mod_python should do fewer things but do them exceptionally well. I would agree with that totally. As a long time user of mod_python, I would add that I personally disagree with including Sessions, psp, publisher, etc. in the base distribution, since these things seem to have held up releases. I don't use any of those things as all with mod_python, and I don't see how they're necessary to use mod_python for what it's for: creating apache extension modules in Python. The thing we need to address is what to do with nifty things we create but that don't qualify for inclusion. The idea of a 'contrib' directory has been floated around for a while, I for one am against it for the same reasons above - it should either be 100% supported or not included at all IMO. If that's the case, there definitely needs to be a central place for contributed code like this. I think a contrib directory is convenient, but I also know the issues of who decides what to put in there, how much do put include, how do you gently waive off support requests for its contents. But there needs to be something. What do you see on the list all the time? People asking how do I implement database pooling, how to I use sessions, here's my database implementation for sessions. I say put the publisher there as well, since it doesn't have anything to do with the philosophy of mod_python, even by Grisha's definition, and solves some release issues with mod_python itself (can anyone say imports?). Nick
Re: Errors in latest beta
Never mind; I think I've tracked down the problem, and I think it's related to the fact that I inadvertently upgraded the python2.4 package on my system to version 2.4.2rc1 :-/ Nick Nick wrote: For some reason I can no longer execute a function registered with req.register_cleanup: [Sat Sep 24 11:56:18 2005] [error] [client 127.0.0.1] exceptions.UnboundLocalError: 1124096352 [Sat Sep 24 11:56:18 2005] [error] [client 127.0.0.1] python_cleanup: Error calling cleanup object 0x2db1f758> I haven't had time to dig into the mod_python code yet, and I'll be somewhat busy this weekend, but I want to give a heads up. I don't rule out the possiblity that it's because of my code ;-) It did worked before I upgraded, though. Basically I'm calling like so: req.register_cleanup(pse_cleanuphandler, req) Nick
Re: SQLite and other things [was Re: svn commit: r290569]
Jim Gallacher wrote: > Nicolas Lehuen wrote: >> I thought that all this mptest.py thing was a bit disturbing, as >> usually people took the wrong impression that they had to call the >> /test/mptest.py URL, that is they thought that the handler system was >> a bit like the publisher system. So by providing a test handler in the >> box, we could make sure that the installation was correct without >> having to cope with all the fuss about where mptest.py should be >> located and what URL to use to make sure everything is working. > > I think this is a worthwhile addition as it should make it easier to get > new people started with mod_python. Looking at this file, it's a great way to verify that mod_python is working correctly, and it's pretty thorough in checking the apache internals. However, it's neither simple nor intuitive for new people to get started with mod_python I think. For a newbie, it looks like it could be overwhelming. Nick
Re: [jira] Created: (MODPYTHON-82) mod_python.publisher cache will not work if threading not built into Python
Instead of the code: from threading import Lock It should use: try: from threading import Lock except: class Lock: def acquire(self): pass def release(self): pass I suggest: try: from threading import Lock except: from dummy_threading import Lock Nick
Re: glue between apache and python logging
Jim Gallacher wrote: Nic Ferrier wrote: Hello mod_python developers. I just joined this list (at the suggestion of Graham Dumpleton) to try and get you guys to consider adding some glue to connect python >2.2 logging to Apache's logging. This means adding a small extra source file to the mod_python codebase. Here's my example: [snip] If the python logging module is meant to be the standard for logging then I'm not against it. However I'm not sure it'll be as simple as Nic's code indicates. I don't want to expand on my concerns until I have a chance to do some testing but in the mean time perhaps Nic could give an example of his usage? (This is a trick question. ;) ) I have done this before, and although I haven't checked this code, it is as trivial to do as Nic makes it appear (my code was a bit different, but it did essentially the same thing). On the other hand, I'm not sure including this code fits with Grisha's philosophy of mod_python being what it is -- glue between Python and apache -- especially when this is such a simple exercise to anyone who has read the logging docs. Nick
Re: glue between apache and python logging
Nic Ferrier wrote: Nick <[EMAIL PROTECTED]> writes: Nic Ferrier wrote: I just joined this list (at the suggestion of Graham Dumpleton) to try and get you guys to consider adding some glue to connect python >2.2 logging to Apache's logging. I have done this before, and although I haven't checked this code, it is as trivial to do as Nic makes it appear (my code was a bit different, but it did essentially the same thing). On the other hand, I'm not sure including this code fits with Grisha's philosophy of mod_python being what it is -- glue between Python and apache -- especially when this is such a simple exercise to anyone who has read the logging docs. But that's my point Nick, it's so simple as to be annoying when you have to do it (it certainly annoyed me when I came to mod_python). I agree, there are several bits of code that are like that. I really think the argument about mod_python being minimal does not apply here. Nick said it - mod_python is glue between Python and apache - well, making Python logging work directly to Apache seems like important glue to me. And again I renew my argument that there needs to be some kind of contrib archive that is probably separate from mod_python and unsupported in the core distribution. Maybe a wiki or code repository or something to support all the contributions like this. That way all these neat things can be available to people who are using mod_python, but not burden the development process of mod_python itself with issues concerning the constributions. Nick
Re: glue between apache and python logging
Nic Ferrier wrote: Nic Ferrier wrote: And again I renew my argument that there needs to be some kind of contrib archive that is probably separate from mod_python and unsupported in the core distribution. Maybe a wiki or code repository or something to support all the contributions like this. That way all these neat things can be available to people who are using mod_python, but not burden the development process of mod_python itself with issues concerning the constributions. I agree. But I don't think logging glue should go in it. logging glue should be in core mod_python because it is glue, not a feature built on top of Apache or Python, simply a link between the two worlds. Maybe I just don't have enough exposure, but I don't see the logging module used as ubiquitously as you seem to imply. I could agree with you if logging was used everywhere, even in Python and its standard modules. I could be wrong, but I just don't see it. If logging is just available in an addon then developers have still got the problem of deploying mod_python PLUS pretty much essential stuff from so and so archive as well as the app. Well, while you may see interfacing with the logging module as "essential" for your work, I just don't see it as a general case. In my case, I ultimately decided not to use the logging module because it was more cumbersome than I needed my logging to be. I wanted to be light and flexible, and anyone using my stuff could implement a logging handler to use my interfaces, if they liked. Call it a difference of opinion until someone can change my mind :) Nick
Re: glue between apache and python logging
Nic Ferrier wrote: But it's difficult to change your mind if you say "prove that logging is the most widely used logging available and then I'll think about it but I don't use it anyway because I wrote my own". Well, that's not what I said. What I said was: 1. I'm not convinced that a logger handler should be included in mod_python just because it's a standard module. A lot of packages out there that have logging facilities don't necessarily use logger. Even though it's part of the Python distribtion, that doesn't make it uniquitous in its use. mod_python exposes apache's logging facility in using interface that apache provides. I think that's the goal of mod_python, to (1) provide access to apache's API and (2) provide dispatching for handlers. 2. My counterargument to the logging interface as being "essential" is that I neither use the logging module nor want to use it. I tried to use it, but the implementation required a lot of overhead (mostly due to threading issues and access to the right req object). YMMV, but my experience led me to create a thinner wrapper around apache.log_error that was simply easier to use and maintain. That really has nothing to do with whether or not In terms of changing my mind, that has to do with whether or not I believe it should be part of the core mod_python distribution. And I still think it should not. I still think it falls under the umbrella of contributed code, just as if it was an implementation that utilized cgitb or SimpleXMLRPCServer, both of which are standard modules and arguably pretty darn useful in mod_python development. Please note that I also disagree with including the publisher and psp handlers as part of the core distribution, but, well, we can't turn back the clock on that ;-) And besides, I'm probably one of the few here who don't use them, making my opinion on that matter a minority view. Unfortunately I have to argue from the position that logging is officially part of the python language. Respectfully, I don't think that has anything to do with it. There's a lot that can be done with only the official parts of the Python language that aren't included in mod_python but left as an option for the user to implement. Recently a bid to include a WSGI implementation for mod_python was shot down, which is a Python PEP and something I thought would have been pretty neat to include. Regardless of Grisha's personal feelings on that (and I have a pretty good idea of what they are ;-)), I think it was ultimately the right decision for mod_python. Nick
Re: glue between apache and python logging
Jim Gallacher wrote: Beyond that it still segfaults for me. The other problem is that you are not removing the handler instance from the logging instance so you still have a memory leak. 100k requests would result in 100k handler instances. Oh, and there might be a bit of a performance issue when emit() gets fired 100k times. ;) Or should I assume that the user will still be required to include req.register_cleanup(log.removeHandler, hdlr) in their code? Not sure if this will help, but in my implementation I registered my hander on the root logger as global code in the handler module, not for each request. In that case I used a threading.local to keep track of req, which I had to register and free for each request of course. I couldn't get around the minimal bookkeeping required of registering the req object on each request, though like Nic's code, I registered a cleanup in a closure to handle the freeing. Alternatively, you can register the server and use apache.log_error with a server object, which should not leak. Also, if you don't care about logging to a particular server, you can, of course, just call apache.log_error without a server argument. Nick
Re: glue between apache and python logging
Graham Dumpleton wrote:
Hopefully everyone follows what I am talking about. I will try and get
together a working example today of what I am talking about, but Nick,
you may want to consider posting your code and how you are using it
as it probably will not be too different.
Here's my sample code I was using. Very rough and unpolished, since I
ultimately decided not to use it. I haven't tried it in a while, but it
should more or less work. apache.py might contain the following code:
from logger import log_handler
Sample usage might look like the following (to steal your example from a
subsequent email):
from mod_python import apache
import logging
log = logging.getLogger("mod_python")
log.warning("importing handler")
def handler(req):
apache.log_handler.set_req(req) # <-- here's the extra code
log.warning("running handler")
req.content_type = 'text/plain'
req.send_http_header()
req.write('hello\n')
return apache.OK
Notice that if you *don't* call set_req, logging will still work, but
not according to any special ErrorLog settings in your apache config
other than the global setting. Also, since log_handler is available,
you can change it's default formatting options somehow as a mod_python
directive in your apache config, or whatever else you want to set.
Nick
from mod_python import apache
import logging, threading
class mpHandler(logging.Handler):
apache_levels = { logging.CRITICAL : apache.APLOG_CRIT,
logging.DEBUG : apache.APLOG_DEBUG,
logging.ERROR : apache.APLOG_ERR,
logging.FATAL : apache.APLOG_EMERG,
logging.INFO : apache.APLOG_INFO,
logging.NOTSET : apache.APLOG_INFO,
logging.WARN : apache.APLOG_WARNING }
def __init__(self, level = logging.NOTSET):
logging.Handler.__init__(self, level)
self.local = threading.local()
self.local.req = None
def emit(self, record):
if self.local.req:
self.local.req.log_error(self.format(record),
mpHandler.apache_levels[record.levelno] | apache.APLOG_NOERRNO)
else:
apache.log_error(self.format(record),
mpHandler.apache_levels[record.levelno] | apache.APLOG_NOERRNO)
def set_req(self, req):
self.local.req = req
def free_req():
self.local.req = None
req.register_cleanup(free_req)
log = logging.getLogger('')
log_handler = mpHandler()
log_handler.setFormatter(logging.Formatter(logging.BASIC_FORMAT))
log.addHandler(log_handler)
log.setLevel(logging.NOTSET)
Re: glue between apache and python logging
Graham Dumpleton wrote:
Yes, effectively the same as what I was doing. As I highlighted in prior
email though about request cache implementations, not sure it would
work correctly if an internal redirect occurred and both original handler
and target of internal redirect had registered request object. One needs
to use a stack for self.local.req and push/pop the request object on to
it.
Hm, very well. I never really considered the case of using the internal
redirect. I'm an "old school" mod_python user, so I'm not up on all the
new features apache2 brings to the table. Reading the docs, I would
suppose you could make use of req.prev somehow in these cases rather
than manually keeping a separate stack? E.g. in my free_req function,
do "self.local.req = self.local.req.prev" (assuming if there hasn't been
an internal_redirect, req.prev is None).
Your code would still work, but if anything was logged by the original
handler after the internal redirect had returned, the request object will
have been wiped out and it would not log through "req" but through
"apache.log_error()" instead.
Right, that's the intention: provide a reasonable fallback.
Graham
Nick wrote ..
Graham Dumpleton wrote:
Hopefully everyone follows what I am talking about. I will try and get
together a working example today of what I am talking about, but Nick,
you may want to consider posting your code and how you are using it
as it probably will not be too different.
Here's my sample code I was using. Very rough and unpolished, since I
ultimately decided not to use it. I haven't tried it in a while, but it
should more or less work. apache.py might contain the following code:
from logger import log_handler
Sample usage might look like the following (to steal your example from
a
subsequent email):
from mod_python import apache
import logging
log = logging.getLogger("mod_python")
log.warning("importing handler")
def handler(req):
apache.log_handler.set_req(req) # <-- here's the extra code
log.warning("running handler")
req.content_type = 'text/plain'
req.send_http_header()
req.write('hello\n')
return apache.OK
Notice that if you *don't* call set_req, logging will still work, but
not according to any special ErrorLog settings in your apache config
other than the global setting. Also, since log_handler is available,
you can change it's default formatting options somehow as a mod_python
directive in your apache config, or whatever else you want to set.
Nick
Re: glue between apache and python logging
Graham Dumpleton wrote: Graham Dumpleton wrote: Unfortunately you can't do that as all Python request objects in that chain are created fresh for the handler which is the target of the internal redirect. When the internal redirect returned, the cached would then be actually different to the original. It would still return the same underlying Apache data, but any extra stuff added to the request object from Python code wouldn't be accessible. The cleanup routine gets called at the end of each request, so I don't think that's an issue that couldn't be resolved with the method I was describing. Perhaps I was being too simplistic in my example to fully explain what I was thinking, though. But in any case, I think the problem could be solved much more simply by changing the following code: def set_req(self, req): prev_req = self.local.req self.local.req = req def free_req(): self.local.req = prev_req req.register_cleanup(free_req) That should take care of restoring the previous "cached" request object without explicitly creating a stack (i.e. relying on the Python call stack to bind local variables in the closure). Nick
Re: glue between apache and python logging
Graham Dumpleton wrote: The strange stuff with sys.modules stuff was to protect against any problems that can occur if people decided to take the sample code and put it straight in their document tree somewhere where it could be reloaded. You'll have to trust me that module reloading can cause some nasty and hard to find problems. :-) Yeah, I know too well the complications of managing imports. What I was really referring to though is what you hit on with this: As to providing "handler()" this is again because it is a separate sample distinct from mod_python. If a part of mod_python that can all be done transparently. I didn't want to provide a sample where people had to hack on their mod_python installation to try it. It seems like there should be a simpler mechanism to be able to use this code without using a separate, second handler. That's confusing to the unitiated. I guess I like the idea of providing a way for you to intialize the logging in your handler code rather than relying on another handler to do it, even though your way makes it somewhat automagic. I prefer things being explicit and simply understood rather than a "just set it up that way to make it work" explanation. In my world, I'd rather see a call to logging.info, for example, raise an exception because I didn't tell it what req to use in some sort of initialization code. I don't think that's altogether unreasonable to ask of the user if they want to use the logging module. Obviously, since my sample implementation worked that way :) But just purely my opinion. Not knocking your code -- I think it accomplishes what you set out to do and does it well. I'd just like to see it done in such a way that is more easily understood by noobs (and subsequently leads to fewer support requests on the list) :) Nick
Re: mod_python 3.2.3b available for testing
Right, that's exactly what I'm having to do; I was thinking though that mod_python should present a consistent interface, even if Python doesn't. And, it is a bug in Python, even if it's a documentation bug (which claims that the behavior of fdopen is to return a file object). I disagree that documenting this fact in mod_python without changing the code doesn't amount to saying you don't support Windows. It just means you support Windows to the extent that Python itself does. Nick Indrek Järve wrote: This behaviour has been with Python for quite a while, so claiming it's simply a Python bug will be the same as declaring we don't support Windows. Our company's software that runs on Windows and uses mod_python simply patches util.py with the following change: 227c227 < if isinstance(item.file, FileType): --- > if isinstance(item.file, FileType) or (hasattr(item.file, 'file') and isinstance(item.file.file, FileType)): I haven't tried this with mp32 yet (we're still running on Python 2.3 and I haven't had time to investigate how to compile mp on Windows), but on 3.0/3.1 it appears to work just fine for our customers. Best regards, Indrek Järve Inversion Software OÜ Nick wrote: More info: python 2.4.2 on Linux: >>> import tempfile >>> t = tempfile.TemporaryFile() >>> t ', mode 'w+b' at 0xb7df07b8> >>> type(t) >>> dir(t) ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 'xreadlines'] python 2.4.1 on windows: >>> import tempfile >>> t = tempfile.TemporaryFile() >>> t ', mode 'w+b' at 0x0099FBA8> >>> type(t) >>> dir(t) ['__doc__', '__getattr__', '__init__', '__module__', '__repr__', 'close_called', 'file', 'name'] So this is an inconsistency within Python. Should mod_python attempt to correct it, or just claim a Python bug? Nick Nick wrote: This may be a Python Windows thing, but it shows up in mod_python: When using util.FieldStorage on multipart/form-data encoded POST data containing a file, in Linux a field.file will yield a file object (actually a subclass of file), but in Windows you have to get the file object through field.file.file. This probably has something to do with the fact that Windows' implementation of tempfile.TemporaryFile is different from Linux, but it should be made consistent in the mod_python interface. Nick Jim Gallacher wrote: A new mod_python 3.2.3 beta tarball is now available for testing. A Windows binary is also available. This release is similar to 3.2.2b but fixes a couple a small issues where a non-threaded python is used. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.3b available for testing
Jim Gallacher wrote: So this is an inconsistency within Python. Should mod_python attempt to correct it, or just claim a Python bug? I think we should correct it. I'm sure users don't care that we implement this with TemporaryFile. That being said, I wonder how many applications on Windows we may break by fixing this? Version 3.1.4 also used TemporaryFile, so this is not a new bug. Yeah, I never noticed it either until someone pointed it out to me. I appreciated the change to TemporaryFile, but being primarily a Linux user I never noticed that this broke my code in Windows. In any case, I'm still gonna have to implement a workaround in my own code to catch people using the different versions of mod_python out there, so I can live with whatever decision you guys make. But here's +1 for making the interface consistent at least for mod_python users. As for code breakage, I would consider this a "bug" introduced in 3.1.4, which was the last official release of mod_python, which will be corrected in release 3.3. Nick
Re: mod_python 3.2.3b available for testing
Jorey Bump wrote:
Are you sure there is anything to correct? In both cases, the object has
the same methods available for manipulating files (t.write('a'), for
example). They are not the same type of object, so they have different
dir() output, but don't they have the same functionality? What
specifically gets broken in util.FieldStorage?
At a minimum, the documentation:
"This is a file object. For file uploads it points to a temporary file. For
simple values, it is a StringIO object, so you can read simple string values
via this attribute instead of using the value attribute as well."
As I stated, it just may be enough to document here that it's a file -or- a
file-type object. That's what the Python documentation for
tempfile.TemporaryFile states, so maybe calling it a "bug" is wrong. But
it's equally simple to return the actual file object, and in my opinion more
convenient for programmers to do isinstance(field.file, file) to see if you
have an uploaded file vs. some other kind of value. I don't know enough
about Windows though to know what will happen if you garbage collect the
tempfile._TemporaryFileWrapper object but keeping a reference to the file.
It's unfortunate that it's not possible to subclass a builtin type and
overload its methods (that I know of).
Nick
Re: mod_python 3.2.3b available for testing
Well, here's another alternative: provide some other attribute to Field,
such as is_file, to determine whether or not the Field is an actual file
upload or something else. Because as implemented, the file attribute will
always return a file-type object.
Nick
Nick wrote:
Jorey Bump wrote:
Are you sure there is anything to correct? In both cases, the object
has the same methods available for manipulating files (t.write('a'),
for example). They are not the same type of object, so they have
different dir() output, but don't they have the same functionality?
What specifically gets broken in util.FieldStorage?
At a minimum, the documentation:
"This is a file object. For file uploads it points to a temporary file.
For simple values, it is a StringIO object, so you can read simple
string values via this attribute instead of using the value attribute
as well."
As I stated, it just may be enough to document here that it's a file
-or- a file-type object. That's what the Python documentation for
tempfile.TemporaryFile states, so maybe calling it a "bug" is wrong.
But it's equally simple to return the actual file object, and in my
opinion more convenient for programmers to do isinstance(field.file,
file) to see if you have an uploaded file vs. some other kind of value.
I don't know enough about Windows though to know what will happen if you
garbage collect the tempfile._TemporaryFileWrapper object but keeping a
reference to the file.
It's unfortunate that it's not possible to subclass a builtin type and
overload its methods (that I know of).
Nick
Re: mod_python 3.2.3b available for testing
Jim Gallacher wrote: You may have misunderstood. I was not suggesting that tempfile.TemporaryFile was introduced in 3.1.4, only that it existed there. Looking at the svn repository I see it's used in 3.0.0-beta and 2.7.9, so this bug has been lurking for a while. ;) Yes, although the fact that the implementation of TemporaryFile changed in Python 2.3 may have something to do with it. I honestly don't remember what the previous behavior was, but it worked OK for me at one time :) Nick
Re: mod_python 3.2.3b available for testing
Jim Gallacher wrote:
Are you sure there is anything to correct? In both cases, the object
has the same methods available for manipulating files (t.write('a'),
for example). They are not the same type of object, so they have
different dir() output, but don't they have the same functionality?
What specifically gets broken in util.FieldStorage?
No, I'm not sure. Now that I play around with it I'm not sure I
understand the problem at all. Perhaps Nick could elaborate?
[...]
Other than the fact that "isinstance(t, FileType): returns False, I
don't see the problem. Nick?
There's 2 issues. I think a documentation change in mod_python where it
states that a file object is returned to say a file-like object instead
is one. That's easy.
The other is that there isn't an easy way to tell whether what you've
got in the field is a file or not, because no matter what you've got
something that "looks" like a file. Under the current documentation you
could have tried isinstance(field.file, file), but that's clearly not
the right thing to do since TemporaryFile doesn't necessarily return an
actual file object. I accept that, but there needs to be an easy way to
know what type the Field object value is -- string or file. Checking
filename or disposition is obtuse, and there may not necessarily be a
file name even if you get a file.
So that's my problem, or at least that's where the conversation has led
me. Is there an easy way to figure out what you've got other than
process of elimination?
Thanks,
Nick
Re: Field.file issue [was Re: mod_python 3.2.3b available for testing]
Jim Gallacher wrote:
How about this? (Excuse the LateX)
\class{Field} instances have the following attributes:
...
\begin{memberdesc}{file}
This is a file-like object. For file uploads it points to a
\class{TemporaryFile} instance. (For more information see
TemporaryFile in the standard python
\citetitle[http://docs.python.org/lib/module-tempfile.html]{tempfile}
module).
For simple values, it is a \class{StringIO} object, so you
can read simple string values via this attribute instead of using
the \member{value} attribute as well.
\end{memberdesc}
Looks good. Reading this probably answers your question after next.
I guess where I'm getting hung-up is in trying to understand why this is
important. Why do you need to know if it's a file object or a file-like
object?
It's important to know how to handle information coming in from a POST,
because generically I may not know whether a field named "the_file" is
intended as a string file name (from type="text") or a file (from
type="file"). I don't know what the semantics of the form being submitted
are, but I need to do something different with strings and files.
First you're talking about file vs. file-like object, and then file vs
string. When you say string do you mean StringIO or StringField. If it's
just a case of StringField vs any of the file-like objects, why not just
use hasattr(field_thing, 'read')?
Because if the Field is a string value, the file attribute is a StringIO (or
cStringIO), which has a read attribute. hasattr(field_thing, 'read') should
*always* return True no matter what the Field stores (see documentation above).
I don't know what it is about this issue, but every time I look at it I
feel like my IQ has dropped a couple of points. Since this is a limited
resource already I don't want to lose any more of it.
Fair enough. I can check to see if file is a StringIO, or a cStringIO, but
that's assuming that some Python implemention doesn't decide to return one
of those from a tempfile.TemporaryFile call. It's probably not important
enough for most people anyway since they're using publisher or something
similar.
Sorry to have wasted time on this.
Nick
Re: Field.file issue [was Re: mod_python 3.2.3b available for testing]
Just FYI for anyone who's still interested...
The fileno method only exists and returns non-negative integers for real
file streams. So that's the trick.
Nick
Nick wrote:
Jim Gallacher wrote:
How about this? (Excuse the LateX)
\class{Field} instances have the following attributes:
...
\begin{memberdesc}{file}
This is a file-like object. For file uploads it points to a
\class{TemporaryFile} instance. (For more information see
TemporaryFile in the standard python
\citetitle[http://docs.python.org/lib/module-tempfile.html]{tempfile}
module).
For simple values, it is a \class{StringIO} object, so you
can read simple string values via this attribute instead of using
the \member{value} attribute as well.
\end{memberdesc}
Looks good. Reading this probably answers your question after next.
I guess where I'm getting hung-up is in trying to understand why this
is important. Why do you need to know if it's a file object or a
file-like object?
It's important to know how to handle information coming in from a POST,
because generically I may not know whether a field named "the_file" is
intended as a string file name (from type="text") or a file (from
type="file"). I don't know what the semantics of the form being
submitted are, but I need to do something different with strings and files.
First you're talking about file vs. file-like object, and then file vs
string. When you say string do you mean StringIO or StringField. If
it's just a case of StringField vs any of the file-like objects, why
not just use hasattr(field_thing, 'read')?
Because if the Field is a string value, the file attribute is a StringIO
(or cStringIO), which has a read attribute. hasattr(field_thing,
'read') should *always* return True no matter what the Field stores (see
documentation above).
I don't know what it is about this issue, but every time I look at it
I feel like my IQ has dropped a couple of points. Since this is a
limited resource already I don't want to lose any more of it.
Fair enough. I can check to see if file is a StringIO, or a cStringIO,
but that's assuming that some Python implemention doesn't decide to
return one of those from a tempfile.TemporaryFile call. It's probably
not important enough for most people anyway since they're using
publisher or something similar.
Sorry to have wasted time on this.
Nick
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
for myself, I have not had problems on ubuntu 5.10 amd64. Nick Jim Gallacher wrote: Michel Jouvin wrote: Jim, I am not totally surprised... I am afraid this is a platform specific issue as we are running mod_python on Tru64. Something like a 64 bits issue. Does it sound a reasonnable possibility ? I have no idea what may be going on, but that seems as likely as anything else. How to progress in troubleshooting ? Again, no clue. :(. Hopefully some of the bigger brains that hang out around here will chime in. I know Indrek Järve tested 3.2.2b on SuSE Linux 9.2 (x86-64). Perhaps he or someone else with a 64-bit platform could try and reproduce the problem. That would tell us if it's 64-bit related or Tru64 related. I've attached my test script if anyone wants to mess with it. I'm sure I don't need to tell you to *not* run it on a production machine. ;) You'll likely want to change the PAUSE variable to something less than 30 seconds, which is the time between the kill calls. I was testing using qemu, and it needs lots of time for things to happen. usage: ./killchildren <# number of loops> Jim > Michel --On jeudi 24 novembre 2005 17:41 -0500 Jim Gallacher <[EMAIL PROTECTED]> wrote: Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher <[EMAIL PROTECTED]> wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
If you provide say FieldStorage.make_dict that returns a dictionary, then I
don't see why the order of the keys is important when the original list is
still available.
Nick
Nicolas Lehuen wrote:
Hi,
Speaking of ordered dictionary :
http://www.voidspace.org.uk/python/weblog/arch_d7_2005_11_19.shtml#e140
Why is the ordering so important ? I do understand we need to support
multiple values per field name, but I don't see why ordering is
needed.
Regards,
Nicolas
2005/11/28, David Fraser <[EMAIL PROTECTED]>:
Gregory (Grisha) Trubetskoy wrote:
Having looked at the FieldStorage code, I'm guessing the idea was that
you parse fields as they come in and append them to a list. This
preserves the original order of fields, in case it is needed.
I'm not sure that maintaining a dictionary alongside the list is the
right thing to do. It might be, but there are some difficult questions
to answer -e.g. how costly is a sequential search, and is the code
complexity (and fieldstorage code is no picnic to read as it is) worth
the speedup?
Also while it would speed up retrieval, it will slow down the "write"
operation - when a field is added to fieldstorage you now need to
append it to the list, AND check whether it exists in the dictionary,
then add it there as well.
How often do developers access form fields via __getitem__? I noticed
the publisher does not use it - it iterates the list, so nothing would
be gained there.
We do it a lot but we copy it into a different dictionary first to get
exactly the setup we want. But dictionary-style access is a very
obvious, pythonic way to do it.
I have a simple 70-line ordereddict implementation which is derived from
dict and remembers the keys in the order that they were assigned when
iterating through the list, this may be a way to go for this. It just
uses a list of keys internally to remember the order, and otherwise is a
dictionary...
Also, something else to consider - is there a simple programatic
solution that could be documented, e.g. something like
my_fs = util.FieldStorage(req)
dict_fs = {}
dict_fs.update(my_fs)
[have no idea whether this will work :-)]
It may work but still has the potential performance problem since it
loops through the keys and then does a getitem on each key which loops
through them again. Not likely to cause problems for a small number of
arguments but not ideal :-)
and voila - you've got a dictionary based fieldstorage?
Anyway, just a few cents from me.
Grisha
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
Jim Gallacher wrote: Nick wrote: Just one comment. It seems like it would be better just to make add_method inline, since everything else in __init__ is, and it never gets called from anywhere else. add_method? Haha, thanks, I haven't had coffee yet. The add_item method, that is. :) I also like properties, but doesn't that cause a problem if someone chooses to subclass FieldStorage? It could if you didn't realize it was a property. But you can always override a property with another property. Nick
Re: Does 3.2 still support python 2.2?
I'm pretty sure we've had a few discussions about being able to use certain functions and modules because they became available in 2.3, and that's what mod_python was going to require. Like the bsddb database version for your session code, for example. Nick Jim Gallacher wrote: From the 3.2.5b doc: (http://www.modpython.org/live/mod_python-3.2.5b/doc-html/inst-prerequisites.html) 2.1 Prerequisites * Python 2.2.1 or later. Earlier versions of Python will not work. Is this still true or have we dropped support for python < 2.3? Has anybody tested using python 2.2.1? Jim
Re: Does 3.2 still support python 2.2?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 http://article.gmane.org/gmane.comp.apache.mod-python.devel/865 Jim Gallacher wrote: | I figured we had moved on to 2.3, but I wanted to make sure I wasn't | missing something before I changed the docs. I'm not sure if there was a | formal decision on this or everyone just assumed it was true. Perhaps a | pronoucement from Grisha to make it offical? | | If python 2.2 support has been dropped then it needs needs to be | mentioned in changes section of the docs, and the README as well. | | Jim | | Nick wrote: | |> I'm pretty sure we've had a few discussions about being able to use |> certain functions and modules because they became available in 2.3, |> and that's what mod_python was going to require. Like the bsddb |> database version for your session code, for example. |> |> Nick |> |> Jim Gallacher wrote: |> |>> From the 3.2.5b doc: |>> |>> (http://www.modpython.org/live/mod_python-3.2.5b/doc-html/inst-prerequisites.html) |>> |>> |>> 2.1 Prerequisites |>> |>> * Python 2.2.1 or later. Earlier versions of Python will not work. |>> |>> |>> Is this still true or have we dropped support for python < 2.3? Has |>> anybody tested using python 2.2.1? |>> |>> Jim |> |> |> |> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDmj4Cv4zJ7LQ+i84RAh9VAJ9rWpumf/Bdky9NuK0bvX96NHrmQQCeKSDD JAF18Qqe3CvDezgOww9599A= =tlHN -END PGP SIGNATURE-
Re: mod_python 3.2.6b available for testing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 Linux (amd64) Ubuntu Breezy (5.10), apache 2.0.54 (mpm-worker), python 2.4 Jim Gallacher wrote: | A new mod_python 3.2.6 beta tarball is now available for testing. | Nicolas has built windows versions for Python 2.4 and Python 2.3 which | should also be available at www.modpython.org/dist shortly. | | This release is similar to 3.2.5b but fixes a couple of issues - | MODPYTHON-95, 96, 97, 98, 99, 105, 106. | | I think if we get enough +1 votes for this version on python-dev that we | should jump right to a 3.2.6 final release rather than go to a wider | beta release. | | So far we have: | | +1 Linux Debian sid, apache 2.0.55 (mpm-prefork), python 2.3 | +1 Linux Debian sarge, apache 2.0.54 (mpm-worker), python 2.3 | +1 Windows XP, python 2.4 | +1 Windows 2000, python 2.3 | +1 Mac OS X 10.3, Apache 2.0.55, python 2.3 | | Here are the rules: | | In order for a file to be officially announced, it has to be tested by | developers on the dev list. Anyone subscribed to this list can (and | should feel obligated to :-) ) test it, and provide feedback *to _this_ | list*! (Not the [EMAIL PROTECTED] list, and preferably not me | personally). | | The files are (temporarily) available here: | | http://www.modpython.org/dist/ | | Please download it, then do the usual | | $ ./configure --with-apxs=/wherever/it/is | $ make | $ (su) | # make install | | Then (as non-root user!) | | $ cd test | $ python test.py | | And see if any tests fail. If they pass, send a +1 to the list, if they | fail, send the details (the versions of OS, Python and Apache, the test | output, and suggestions, if any). | | Thank you, | Jim Ga -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDy83Fv4zJ7LQ+i84RAibhAKCylMw8FP9uvL4WyGQ6VMTyqCh03gCeM3AI vE+fdd6204+kXz2z0rHx1gY= =UjE3 -END PGP SIGNATURE-
Re: please set up a mod_python core group
Jim Gallacher wrote: > Jorey Bump wrote: >> +1 here, but since the build process and typical MPM differs among >> platforms, could we see a list that this group represents? I'm most >> interested in default nonvirtualized environments used in production >> or for principal development. This information will be useful when >> reviewing release candidates, to make sure we haven't overlooked any >> key platforms. > > Your point on making sure we don't overlook any key platforms in our > testing is a good one. Should we (python-dev people) put together a list > of key platforms as a future guide? It's likely a good idea, even at > the risk of a flamewar. ;) I thought I'd put together a summary of 3.2.6 > test results in the next few days anyway, which should be a good > starting point for the key list. As a non-x86 user (amd64 here), I second the notion that we need some non-Linux non-x86 platform testing out there, if people were willing to commit to be available to build and test when that time comes around (I think it's been pretty good, about every 2 months it's been on average?). I know there are people on PPC OSX, FreeBSD, AIX, Tru64, Solaris, and I just think it's a good idea to have a general concensus that a build will work on at least some of these platforms that both apache and Python are also supported and has worked for in the past. I'm not sure which of these you can identify as "key," but I would say that *BSD, OSX and Solaris should top the list. I also suggest Linux x86_64 of some kind, since it's becoming more and more widely used; I know we've got 2 or 3 people that normally respond to release tests that do. Nick
Re: mod_python 3.2.7 available for testing
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 +1 Ubuntu 5.10 Breezy (amd64), Apache 2.0.54-worker, Python 2.4.2 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFD5/mTv4zJ7LQ+i84RAofQAKCb4ptmhPQa5QKRV/2sga60Xz4oAACcDygf IB8UDE0zlcUr+I16DWbQ09U= =WrUY -END PGP SIGNATURE-
