First impressions would suggest that request handler threads are blocking on some operation and simply never completing. Eventually what happens is that there is a period long enough that the inactivity timeout is kicking in and the daemon process is shutdown. When this occurs the connection from Apache child process for the blocked request is broken and the message 'Premature end of script headers: django.wsgi' results.
I would suggest integrating some variant of: http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces into your application such that you have a means of triggering a dump of what all request handler threads are doing. By luck you may be able to trigger this when there are some of these blocked threads and work out what they are blocking on. Since your inactivity timeout is an hour long, then that may not be too hard actually. Set up a cron job to trigger the stack dump every 20 minutes and see if you see request threads that are always stuck at same point. Alternative to cron is just modify that example to do a sleep() within the background thread itself and just automatically dump out every 20 minutes. Graham On 2 May 2011 11:08, Paddy Joy <[email protected]> wrote: > Hi, > > Hope someone can help debug the following issue I am having. > > I have been running approx 15 django sites with mod_wsgi for a few > years now with no problems. In the last month I am getting sites that > randomly stop working, the process is gone and apache returns 500 > status. I have set logging level to info but it still doesn't give me > enough info to pinpoint the problem. The only notable change I made to > the server was a switch to apache mpm-itk a couple of months back. > > Server config is as follows, example of django.wsgi and virtual host > at bottom of email. > apache 2.2.8 mpm-itk > python 2.5.2 > mod_wsgi 3.3 daemon mode > mod_python not installed > > Example: > > apache access log, site ok at 2am when googlebot visited then 500 > error in morning > 66.249.71.166 - - [02/May/2011:02:17:11 +1000] "GET /password_reset/ > HTTP/1.1" 200 6071 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; > +http://www.google.com/bot.html)" > 120.156.100.6 - - [02/May/2011:09:08:27 +1000] "GET /favicon.ico HTTP/ > 1.1" 500 739 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) > Gecko/20100914 Firefox/3.6.10" > > Error log: > [Mon May 02 02:17:11 2011] [info] mod_wsgi (pid=22081): Create > interpreter 'www.builderassist.com.au|'. > [Mon May 02 02:17:11 2011] [info] [client 66.249.71.166] mod_wsgi > (pid=22081, process='buildassist', > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > django/buildassist/django.wsgi'. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Daemon process > inactivity timer expired, stopping process 'buildassist'. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Shutdown > requested 'buildassist'. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Stopping > process 'buildassist'. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroying > interpreters. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroy > interpreter 'www.builderassist.com.au|'. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Cleanup > interpreter ''. > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Terminating > Python. > [Mon May 02 03:17:14 2011] [info] mod_wsgi (pid=22081): Python has > shutdown. > [Mon May 02 09:13:27 2011] [error] [client 120.156.100.6] Premature > end of script headers: django.wsgi > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > end of script headers: django.wsgi > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > end of script headers: django.wsgi > ---relaod apache here > [Mon May 02 10:21:26 2011] [info] mod_wsgi (pid=499): Attach > interpreter ''. > [Mon May 02 10:21:28 2011] [info] mod_wsgi (pid=499): Create > interpreter 'www.builderassist.com.au|'. > [Mon May 02 10:21:28 2011] [info] [client 120.156.100.6] mod_wsgi > (pid=499, process='buildassist', > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > django/buildassist/django.wsgi'. > > I am getting the same thing with the other sites, happens about 3 or 4 > times a week and requires apache reload to fix. > > Any ideas? > > Thanks, > Paddy > > > ----- virtual host > -------------------------------------------------------------- > > WSGIDaemonProcess buildassist user=django group=django threads=25 > display-name=%{GROUP} inactivity-timeout=3600 > WSGIProcessGroup buildassist > > WSGIScriptAlias / /var/django/buildassist/django.wsgi > > > ----- django.wsgi > --------------------------------------------------------------- > > ALLDIRS = ['/usr/local/pythonenv/DJANGO_LEGACY_1.2/lib/python2.5/site- > packages/'] > APP_NAME = 'buildassist' > import os, sys, site > > # virtualenv stuff ------------------------------ > prev_sys_path = list(sys.path) > > for directory in ALLDIRS: > site.addsitedir(directory) > > new_sys_path = [] > > for item in list(sys.path): > if item not in prev_sys_path: > new_sys_path.append(item) > sys.path.remove(item) > sys.path[:0] = new_sys_path > > # -------------------------------- > > sys.path.append('/var/django') > sys.path.append('/var/django/%s' % APP_NAME) > > os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % APP_NAME > os.environ['PYTHON_EGG_CACHE'] = '/tmp/eggs' > > import django.core.handlers.wsgi > > _application = django.core.handlers.wsgi.WSGIHandler() > > import posixpath > > def application(environ, start_response): > # Wrapper to set SCRIPT_NAME to actual mount point. > environ['SCRIPT_NAME'] = posixpath.dirname(environ['SCRIPT_NAME']) > if environ['SCRIPT_NAME'] == '/': > environ['SCRIPT_NAME'] = '' > return _application(environ, start_response) > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
