Graham, Thanks for your prompt help as usual. I have installed the code in one of the applications so hopefully I can catch the timeout. It's strange because it is different urls on different sites that seems to start the issue, there seems to be no pattern. Some of the code on the older sites hasn't been touched in months.
When the daemon process reaches the timeout and is shutdown, shouldn't a new process be started the next time a request comes in? I need to put something in place for the short term so that I can reload apache once this situation occurs, should I use a simple script to monitor if each application still has a process running or what would you recommend? Paddy On May 2, 11:19 am, Graham Dumpleton <[email protected]> wrote: > First impressions would suggest that request handler threads are > blocking on some operation and simply never completing. Eventually > what happens is that there is a period long enough that the inactivity > timeout is kicking in and the daemon process is shutdown. When this > occurs the connection from Apache child process for the blocked > request is broken and the message 'Premature end of script headers: > django.wsgi' results. > > I would suggest integrating some variant of: > > http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_... > > into your application such that you have a means of triggering a dump > of what all request handler threads are doing. By luck you may be able > to trigger this when there are some of these blocked threads and work > out what they are blocking on. Since your inactivity timeout is an > hour long, then that may not be too hard actually. Set up a cron job > to trigger the stack dump every 20 minutes and see if you see request > threads that are always stuck at same point. Alternative to cron is > just modify that example to do a sleep() within the background thread > itself and just automatically dump out every 20 minutes. > > Graham > > On 2 May 2011 11:08, Paddy Joy <[email protected]> wrote: > > > Hi, > > > Hope someone can help debug the following issue I am having. > > > I have been running approx 15 django sites with mod_wsgi for a few > > years now with no problems. In the last month I am getting sites that > > randomly stop working, the process is gone and apache returns 500 > > status. I have set logging level to info but it still doesn't give me > > enough info to pinpoint the problem. The only notable change I made to > > the server was a switch to apache mpm-itk a couple of months back. > > > Server config is as follows, example of django.wsgi and virtual host > > at bottom of email. > > apache 2.2.8 mpm-itk > > python 2.5.2 > > mod_wsgi 3.3 daemon mode > > mod_python not installed > > > Example: > > > apache access log, site ok at 2am when googlebot visited then 500 > > error in morning > > 66.249.71.166 - - [02/May/2011:02:17:11 +1000] "GET /password_reset/ > > HTTP/1.1" 200 6071 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; > > +http://www.google.com/bot.html)" > > 120.156.100.6 - - [02/May/2011:09:08:27 +1000] "GET /favicon.ico HTTP/ > > 1.1" 500 739 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) > > Gecko/20100914 Firefox/3.6.10" > > > Error log: > > [Mon May 02 02:17:11 2011] [info] mod_wsgi (pid=22081): Create > > interpreter 'www.builderassist.com.au|'. > > [Mon May 02 02:17:11 2011] [info] [client 66.249.71.166] mod_wsgi > > (pid=22081, process='buildassist', > > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > > django/buildassist/django.wsgi'. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Daemon process > > inactivity timer expired, stopping process 'buildassist'. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Shutdown > > requested 'buildassist'. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Stopping > > process 'buildassist'. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroying > > interpreters. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroy > > interpreter 'www.builderassist.com.au|'. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Cleanup > > interpreter ''. > > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Terminating > > Python. > > [Mon May 02 03:17:14 2011] [info] mod_wsgi (pid=22081): Python has > > shutdown. > > [Mon May 02 09:13:27 2011] [error] [client 120.156.100.6] Premature > > end of script headers: django.wsgi > > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > > end of script headers: django.wsgi > > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > > end of script headers: django.wsgi > > ---relaod apache here > > [Mon May 02 10:21:26 2011] [info] mod_wsgi (pid=499): Attach > > interpreter ''. > > [Mon May 02 10:21:28 2011] [info] mod_wsgi (pid=499): Create > > interpreter 'www.builderassist.com.au|'. > > [Mon May 02 10:21:28 2011] [info] [client 120.156.100.6] mod_wsgi > > (pid=499, process='buildassist', > > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > > django/buildassist/django.wsgi'. > > > I am getting the same thing with the other sites, happens about 3 or 4 > > times a week and requires apache reload to fix. > > > Any ideas? > > > Thanks, > > Paddy > > > ----- virtual host > > -------------------------------------------------------------- > > > WSGIDaemonProcess buildassist user=django group=django threads=25 > > display-name=%{GROUP} inactivity-timeout=3600 > > WSGIProcessGroup buildassist > > > WSGIScriptAlias / /var/django/buildassist/django.wsgi > > > ----- django.wsgi > > --------------------------------------------------------------- > > > ALLDIRS = ['/usr/local/pythonenv/DJANGO_LEGACY_1.2/lib/python2.5/site- > > packages/'] > > APP_NAME = 'buildassist' > > import os, sys, site > > > # virtualenv stuff ------------------------------ > > prev_sys_path = list(sys.path) > > > for directory in ALLDIRS: > > site.addsitedir(directory) > > > new_sys_path = [] > > > for item in list(sys.path): > > if item not in prev_sys_path: > > new_sys_path.append(item) > > sys.path.remove(item) > > sys.path[:0] = new_sys_path > > > # -------------------------------- > > > sys.path.append('/var/django') > > sys.path.append('/var/django/%s' % APP_NAME) > > > os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % APP_NAME > > os.environ['PYTHON_EGG_CACHE'] = '/tmp/eggs' > > > import django.core.handlers.wsgi > > > _application = django.core.handlers.wsgi.WSGIHandler() > > > import posixpath > > > def application(environ, start_response): > > # Wrapper to set SCRIPT_NAME to actual mount point. > > environ['SCRIPT_NAME'] = posixpath.dirname(environ['SCRIPT_NAME']) > > if environ['SCRIPT_NAME'] == '/': > > environ['SCRIPT_NAME'] = '' > > return _application(environ, start_response) > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/modwsgi?hl=en. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
