David Fraser wrote:
David Fraser wrote:Graham Dumpleton wrote:On 18/08/2006, at 7:48 PM, Richard Lewis wrote:On Tuesday 15 August 2006 23:31, Graham Dumpleton wrote:Richard Lewis wrote ..Do you mean that there will be no opportunity to have code run at server shutdown?Correct. Reason being that it doesn't actually work most of the time anyway.Because of how Apache is implemented, there is no reliable/safe way of implementing this feature. If one can't do it properly, it seems better not to attempt it at all.Goodness! So, in my mod_python applications I often acquire database handles and store them in objects outside of the handler() function so that they persist between requests. (This is to avoid the expense of acquiring a new handle for every request).And you can still do that.I then use a cleanup() function to release those database handles. The implication of this is that I will no longer be able to release the handles at server shutdown, yes?I think you miss what I have been saying. That it doesn't work now. If you were to put calls to apache.log_error() in your cleanup handler and shutdown Apache when it is handling a decent amount of traffic, you will probably find your cleanup functions aren't actually being called, or if they do, they might not exit and will hang at some point. What you are more likely to see in the Apache log is a series of SIGTERM signals being sent and then a SIGKILL signal which is forcibly killing off the process.Can anyone suggest an alternative method of doing this? Are there any Python tricks where you can execute code when the interpreter itself is about to stop?Python does have means of calling code on application shutdown, but because of how Apache uses signals to shutdown processes and how the Apache main loop works, with the main loop being managed by Apache and not Python they can't be used either.Or could I have a Python script running in another process which "looks in" to the mod_python process and periodically cleanly releases database handles?No. In the greater scheme of things it shouldn't ultimately matter. This is because your database server is going to notice that the connections to it have been dropped and will cleanup the resources on its side anyway. It has to do this as there is nothing to say that Apache or any client process connecting to it will not simply crash without cleanly disconnecting. In other words, you will not get resource leaks. That things didn't get cleanup in the Apache child process doesn't matter as it has been killed anyway, with all its memory beingreleased back to operating system.So you are saying: 1) There is a mechanism for cleaning up code 2) This mechanism is not reliable 3) Since databases have to assume clients are not reliable, they clean up for them anyway 4) Therefore we should not even try to clean up I'm with you on points 1, 2, and 3, but I think point 4 is taking it a bit too far... Surely there must be *some* value in trying to clean up behind yourself, sometimes?Hi I thought it would be good to take this across to python-dev. I've read through https://issues.apache.org/jira/browse/MODPYTHON-109?page=all and the discussion in http://www.modpython.org/pipermail/mod_python/2006-January/019865.html http://www.modpython.org/pipermail/mod_python/2006-January/019866.html and http://www.modpython.org/pipermail/mod_python/2006-January/019870.html again, and I'm just not sure about this. Basically, Apache seems to provide some sort of mechanism for child processes to clean themselves up, and for modules to clean up their resources in a particular child. The argument to remove the ability to clean up Python objects seems to be that: A) The finalize method was been called in an awkward place (from inside a signal handler) and other code may be running and have the GIL, so it may not be called at all, even in a graceful shutdown. B) A normal restart will just send a TERM signal, which doesn't give proper opportunity for cleanup C) If the graceful shutdown doesn't work or respond quickly, Apache will just kill the process anyway, so we may as will live with being killed (talk about mixed metaphors...) D) Since databases etc have to deal with the client process being killed, they generally will handle this I accept that problem A with the finalizing methods is a real problem, but wonder if there are alternate solutions that can be provided to allow cleanups to be attempted. I don't think that B or C is a good argument - in that case why would Apache be providing the hooks to clean up anyway? It feels like throwing in the towel... And D just seems impolite - if we can try and clean up we should. Of course, if we can't manage to call finalize methods even in a graceful shutdown none of this may be possible... Trying to find relevant info on this from the Apache docs and other module documentation: http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop talks about advising children to exit after their current request. In this case it would seem the cleanup methods should get called at the end of the request processing, and thus shouldn't be in a signal handler (and there should be no other Python code executing...)
Except that the parent "advises" it's children by sending a signal, doesn't it?
http://www.apachetutor.org/dev/pools talks about using pools to allocate/deallocate resources other than memory - could we provide a way to register Python objects that need cleanup using this mechanism?
That *is* the mechanism that mod_python uses to register cleanups. req.register_cleanup uses the request pool, and apache.register_cleanup uses the server pool (child_init_pool).
Am I barking up the wrong tree or is this worth investigating further? David
It's worth investigating. There may be a solution, but we just can't see it. I don't think anyone would argue that the current proposal to drop the server cleanup is sub-optimal, but the current implementation is worse than having no cleanup at all.
Really though, isn't this whole discussion actually about database connection pooling? Doesn't that cover 99% of the cases people care about? If so maybe our energies would be better focused on what may be required to support mod_dbd within mod_python.
http://httpd.apache.org/docs/2.2/mod/mod_dbd.html Jim
