Thanks Graham I will report back anything I can find. In my situation every time the "premature end of script" occurs the daemon process is not recreated which is a headache with sites dropping offline.
Paddy On May 2, 1:25 pm, Graham Dumpleton <[email protected]> wrote: > The failure to create replacement daemon mode processes is something > that has been seen, but vary rarely, with prefork MPM. It has been an > issue that has been hard to track down as no useful information to > really say what is preceding that. For example, whether there is a > temporary resource issue on the host for example. Right now there is > no indication that it is a mod_wsgi specific issue. So, if you do > manage to capture any evidence of threads being stuck at a particular > place, it may be useful. > > Graham > > On 2 May 2011 12:48, Paddy Joy <[email protected]> wrote: > > > Graham, > > > Thanks for your prompt help as usual. I have installed the code in one > > of the applications so hopefully I can catch the timeout. It's strange > > because it is different urls on different sites that seems to start > > the issue, there seems to be no pattern. Some of the code on the older > > sites hasn't been touched in months. > > > When the daemon process reaches the timeout and is shutdown, shouldn't > > a new process be started the next time a request comes in? I need to > > put something in place for the short term so that I can reload apache > > once this situation occurs, should I use a simple script to monitor if > > each application still has a process running or what would you > > recommend? > > > Paddy > > > On May 2, 11:19 am, Graham Dumpleton <[email protected]> > > wrote: > >> First impressions would suggest that request handler threads are > >> blocking on some operation and simply never completing. Eventually > >> what happens is that there is a period long enough that the inactivity > >> timeout is kicking in and the daemon process is shutdown. When this > >> occurs the connection from Apache child process for the blocked > >> request is broken and the message 'Premature end of script headers: > >> django.wsgi' results. > > >> I would suggest integrating some variant of: > > >>http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_... > > >> into your application such that you have a means of triggering a dump > >> of what all request handler threads are doing. By luck you may be able > >> to trigger this when there are some of these blocked threads and work > >> out what they are blocking on. Since your inactivity timeout is an > >> hour long, then that may not be too hard actually. Set up a cron job > >> to trigger the stack dump every 20 minutes and see if you see request > >> threads that are always stuck at same point. Alternative to cron is > >> just modify that example to do a sleep() within the background thread > >> itself and just automatically dump out every 20 minutes. > > >> Graham > > >> On 2 May 2011 11:08, Paddy Joy <[email protected]> wrote: > > >> > Hi, > > >> > Hope someone can help debug the following issue I am having. > > >> > I have been running approx 15 django sites with mod_wsgi for a few > >> > years now with no problems. In the last month I am getting sites that > >> > randomly stop working, the process is gone and apache returns 500 > >> > status. I have set logging level to info but it still doesn't give me > >> > enough info to pinpoint the problem. The only notable change I made to > >> > the server was a switch to apache mpm-itk a couple of months back. > > >> > Server config is as follows, example of django.wsgi and virtual host > >> > at bottom of email. > >> > apache 2.2.8 mpm-itk > >> > python 2.5.2 > >> > mod_wsgi 3.3 daemon mode > >> > mod_python not installed > > >> > Example: > > >> > apache access log, site ok at 2am when googlebot visited then 500 > >> > error in morning > >> > 66.249.71.166 - - [02/May/2011:02:17:11 +1000] "GET /password_reset/ > >> > HTTP/1.1" 200 6071 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; > >> > +http://www.google.com/bot.html)" > >> > 120.156.100.6 - - [02/May/2011:09:08:27 +1000] "GET /favicon.ico HTTP/ > >> > 1.1" 500 739 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) > >> > Gecko/20100914 Firefox/3.6.10" > > >> > Error log: > >> > [Mon May 02 02:17:11 2011] [info] mod_wsgi (pid=22081): Create > >> > interpreter 'www.builderassist.com.au|'. > >> > [Mon May 02 02:17:11 2011] [info] [client 66.249.71.166] mod_wsgi > >> > (pid=22081, process='buildassist', > >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > >> > django/buildassist/django.wsgi'. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Daemon process > >> > inactivity timer expired, stopping process 'buildassist'. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Shutdown > >> > requested 'buildassist'. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Stopping > >> > process 'buildassist'. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroying > >> > interpreters. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroy > >> > interpreter 'www.builderassist.com.au|'. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Cleanup > >> > interpreter ''. > >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Terminating > >> > Python. > >> > [Mon May 02 03:17:14 2011] [info] mod_wsgi (pid=22081): Python has > >> > shutdown. > >> > [Mon May 02 09:13:27 2011] [error] [client 120.156.100.6] Premature > >> > end of script headers: django.wsgi > >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > >> > end of script headers: django.wsgi > >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > >> > end of script headers: django.wsgi > >> > ---relaod apache here > >> > [Mon May 02 10:21:26 2011] [info] mod_wsgi (pid=499): Attach > >> > interpreter ''. > >> > [Mon May 02 10:21:28 2011] [info] mod_wsgi (pid=499): Create > >> > interpreter 'www.builderassist.com.au|'. > >> > [Mon May 02 10:21:28 2011] [info] [client 120.156.100.6] mod_wsgi > >> > (pid=499, process='buildassist', > >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > >> > django/buildassist/django.wsgi'. > > >> > I am getting the same thing with the other sites, happens about 3 or 4 > >> > times a week and requires apache reload to fix. > > >> > Any ideas? > > >> > Thanks, > >> > Paddy > > >> > ----- virtual host > >> > -------------------------------------------------------------- > > >> > WSGIDaemonProcess buildassist user=django group=django threads=25 > >> > display-name=%{GROUP} inactivity-timeout=3600 > >> > WSGIProcessGroup buildassist > > >> > WSGIScriptAlias / /var/django/buildassist/django.wsgi > > >> > ----- django.wsgi > >> > --------------------------------------------------------------- > > >> > ALLDIRS = ['/usr/local/pythonenv/DJANGO_LEGACY_1.2/lib/python2.5/site- > >> > packages/'] > >> > APP_NAME = 'buildassist' > >> > import os, sys, site > > >> > # virtualenv stuff ------------------------------ > >> > prev_sys_path = list(sys.path) > > >> > for directory in ALLDIRS: > >> > site.addsitedir(directory) > > >> > new_sys_path = [] > > >> > for item in list(sys.path): > >> > if item not in prev_sys_path: > >> > new_sys_path.append(item) > >> > sys.path.remove(item) > >> > sys.path[:0] = new_sys_path > > >> > # -------------------------------- > > >> > sys.path.append('/var/django') > >> > sys.path.append('/var/django/%s' % APP_NAME) > > >> > os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % APP_NAME > >> > os.environ['PYTHON_EGG_CACHE'] = '/tmp/eggs' > > >> > import django.core.handlers.wsgi > > >> > _application = django.core.handlers.wsgi.WSGIHandler() > > >> > import posixpath > > >> > def application(environ, start_response): > >> > # Wrapper to set SCRIPT_NAME to actual mount point. > >> > environ['SCRIPT_NAME'] = posixpath.dirname(environ['SCRIPT_NAME']) > >> > if environ['SCRIPT_NAME'] == '/': > >> > environ['SCRIPT_NAME'] = '' > >> > return _application(environ, start_response) > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "modwsgi" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/modwsgi?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/modwsgi?hl=en. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
