On 14 March 2010 14:57, Alex <[email protected]> wrote: > The vanishing daemons problem seems to be causing an issue on a new > server I've recently deployed. It may have been a very rare > occurrence on an old single core machine (really can't remember), but > now random daemons are disappearing now we've switched to a more > modern quad core xeon server. > > I've tried raising the debug level in Apache to see if it can show any > useful details, a typical working restart: > > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Daemon process > inactivity timer expired, stopping process 'fatfluffs'. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Shutdown > requested 'fatfluffs'. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Stopping > process 'fatfluffs'. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Destroying > interpreters. > [Sun Mar 14 01:14:02 2010] [debug] mod_wsgi.c(5172): mod_wsgi > (pid=6795): Create thread state for thread 0 against interpreter > 'www.fatfluffs.com|/site.wsgi'. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Destroy > interpreter 'www.fatfluffs.com|/site.wsgi'. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Cleanup > interpreter ''. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Terminating > Python. > [Sun Mar 14 01:14:02 2010] [info] mod_wsgi (pid=6795): Python has > shutdown. > [Sun Mar 14 01:14:03 2010] [info] mod_wsgi (pid=6795): Process > 'fatfluffs' has died, restarting. > [Sun Mar 14 01:14:03 2010] [info] mod_wsgi (pid=7134): Starting > process 'fatfluffs' with uid=33, gid=33 and threads=15. > [Sun Mar 14 01:14:03 2010] [info] mod_wsgi (pid=7134): Initializing > Python. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(11151): mod_wsgi > (pid=7134): Process 'fatfluffs' logging to 'www.fatfluffs.com' with > log level 7. > [Sun Mar 14 01:14:03 2010] [info] mod_wsgi (pid=7134): Attach > interpreter ''. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10662): mod_wsgi > (pid=7134): Starting 15 threads in daemon process 'fatfluffs'. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10491): mod_wsgi > (pid=7134): Enable monitor thread in process 'fatfluffs'. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10672): mod_wsgi > (pid=7134): Starting thread 1 in daemon process 'fatfluffs'. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10495): mod_wsgi > (pid=7134): Deadlock timeout is 300. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10498): mod_wsgi > (pid=7134): Inactivity timeout is 300. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10461): mod_wsgi > (pid=7134): Enable deadlock thread in process 'fatfluffs'. > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10672): mod_wsgi > (pid=7134): Starting thread 2 in daemon process 'fatfluffs'. > (removed repeated lines) > [Sun Mar 14 01:14:03 2010] [debug] mod_wsgi.c(10672): mod_wsgi > (pid=7134): Starting thread 15 in daemon process 'fatfluffs'. > [Sun Mar 14 01:14:06 2010] [debug] mod_wsgi.c(11925): mod_wsgi > (pid=7104): Request server was 'www.fatfluffs.com|0'. > [Sun Mar 14 01:14:06 2010] [debug] mod_wsgi.c(12676): mod_wsgi > (pid=7134): Server listener address '|80'. > [Sun Mar 14 01:14:06 2010] [debug] mod_wsgi.c(12685): mod_wsgi > (pid=7134): Server listener address '|80' was found. > [Sun Mar 14 01:14:06 2010] [debug] mod_wsgi.c(12697): mod_wsgi > (pid=7134): Connection server matched was 'drake.hawkz.com|80'. > [Sun Mar 14 01:14:06 2010] [debug] mod_wsgi.c(12713): mod_wsgi > (pid=7134): Request server matched was 'www.fatfluffs.com|0'. > > Then this is the last restart in the logs before it started failing > requests: > > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Daemon process > inactivity timer expired, stopping process 'fatfluffs'. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Shutdown > requested 'fatfluffs'. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Stopping > process 'fatfluffs'. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Destroying > interpreters. > [Sun Mar 14 01:44:07 2010] [debug] mod_wsgi.c(5172): mod_wsgi > (pid=7471): Create thread state for thread 0 against interpreter > 'www.fatfluffs.com|/site.wsgi'. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Destroy > interpreter 'www.fatfluffs.com|/site.wsgi'. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Cleanup > interpreter ''. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Terminating > Python. > [Sun Mar 14 01:44:07 2010] [info] mod_wsgi (pid=7471): Python has > shutdown. > [Sun Mar 14 01:44:08 2010] [debug] mod_wsgi.c(11925): mod_wsgi > (pid=7608): Request server was 'www.fatfluffs.com|0'. > > After this point requests slowly end up with error 500 until an Apache > reload is needed to get the missing daemon back. > > We're running mostly standard Debian Lenny stuff, Apache 2.2.9, tried > most versions of mod_wsgi (packaged, backport, latest) and all seem to > end up with the same problem. WSGIRestrictEmbedded is turned on with > mod_wsgi 3.2, all daemons are standard 1 process, 15 thread, 1000 max > requests, 300 sec timeout. > > It's certainly occurring every few days now and is getting highly > annoying. :( > > If there's any other details needed to figure this one out then let me > know.
And what appears in any error logs for the 500 errors or in the error response page itself? Can you enable multi lang error documents in Apache? When these are used, they will some times display additional error notes about the reason for the error. If for some reason the mod_wsgi daemon process are disappearing, I would not expect to see 500 errors, I would expect to see 503 errors and there would be a series of mod_wsgi specific warning/error messages in Apache error logs resulting from the failures and retries in attempting to connect to the mod_wsgi daemon processes. Graham -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
