Unless you have a real need for it, get rid of the inactivity-timeout option. This will avoid daemon processes restarting and so potentially triggering the issue.
On 2 May 2011 14:10, Paddy Joy <[email protected]> wrote: > Thanks Graham I will report back anything I can find. In my situation > every time the "premature end of script" occurs the daemon process is > not recreated which is a headache with sites dropping offline. > > Paddy > > On May 2, 1:25 pm, Graham Dumpleton <[email protected]> > wrote: >> The failure to create replacement daemon mode processes is something >> that has been seen, but vary rarely, with prefork MPM. It has been an >> issue that has been hard to track down as no useful information to >> really say what is preceding that. For example, whether there is a >> temporary resource issue on the host for example. Right now there is >> no indication that it is a mod_wsgi specific issue. So, if you do >> manage to capture any evidence of threads being stuck at a particular >> place, it may be useful. >> >> Graham >> >> On 2 May 2011 12:48, Paddy Joy <[email protected]> wrote: >> >> > Graham, >> >> > Thanks for your prompt help as usual. I have installed the code in one >> > of the applications so hopefully I can catch the timeout. It's strange >> > because it is different urls on different sites that seems to start >> > the issue, there seems to be no pattern. Some of the code on the older >> > sites hasn't been touched in months. >> >> > When the daemon process reaches the timeout and is shutdown, shouldn't >> > a new process be started the next time a request comes in? I need to >> > put something in place for the short term so that I can reload apache >> > once this situation occurs, should I use a simple script to monitor if >> > each application still has a process running or what would you >> > recommend? >> >> > Paddy >> >> > On May 2, 11:19 am, Graham Dumpleton <[email protected]> >> > wrote: >> >> First impressions would suggest that request handler threads are >> >> blocking on some operation and simply never completing. Eventually >> >> what happens is that there is a period long enough that the inactivity >> >> timeout is kicking in and the daemon process is shutdown. When this >> >> occurs the connection from Apache child process for the blocked >> >> request is broken and the message 'Premature end of script headers: >> >> django.wsgi' results. >> >> >> I would suggest integrating some variant of: >> >> >>http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_... >> >> >> into your application such that you have a means of triggering a dump >> >> of what all request handler threads are doing. By luck you may be able >> >> to trigger this when there are some of these blocked threads and work >> >> out what they are blocking on. Since your inactivity timeout is an >> >> hour long, then that may not be too hard actually. Set up a cron job >> >> to trigger the stack dump every 20 minutes and see if you see request >> >> threads that are always stuck at same point. Alternative to cron is >> >> just modify that example to do a sleep() within the background thread >> >> itself and just automatically dump out every 20 minutes. >> >> >> Graham >> >> >> On 2 May 2011 11:08, Paddy Joy <[email protected]> wrote: >> >> >> > Hi, >> >> >> > Hope someone can help debug the following issue I am having. >> >> >> > I have been running approx 15 django sites with mod_wsgi for a few >> >> > years now with no problems. In the last month I am getting sites that >> >> > randomly stop working, the process is gone and apache returns 500 >> >> > status. I have set logging level to info but it still doesn't give me >> >> > enough info to pinpoint the problem. The only notable change I made to >> >> > the server was a switch to apache mpm-itk a couple of months back. >> >> >> > Server config is as follows, example of django.wsgi and virtual host >> >> > at bottom of email. >> >> > apache 2.2.8 mpm-itk >> >> > python 2.5.2 >> >> > mod_wsgi 3.3 daemon mode >> >> > mod_python not installed >> >> >> > Example: >> >> >> > apache access log, site ok at 2am when googlebot visited then 500 >> >> > error in morning >> >> > 66.249.71.166 - - [02/May/2011:02:17:11 +1000] "GET /password_reset/ >> >> > HTTP/1.1" 200 6071 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; >> >> > +http://www.google.com/bot.html)" >> >> > 120.156.100.6 - - [02/May/2011:09:08:27 +1000] "GET /favicon.ico HTTP/ >> >> > 1.1" 500 739 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) >> >> > Gecko/20100914 Firefox/3.6.10" >> >> >> > Error log: >> >> > [Mon May 02 02:17:11 2011] [info] mod_wsgi (pid=22081): Create >> >> > interpreter 'www.builderassist.com.au|'. >> >> > [Mon May 02 02:17:11 2011] [info] [client 66.249.71.166] mod_wsgi >> >> > (pid=22081, process='buildassist', >> >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ >> >> > django/buildassist/django.wsgi'. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Daemon process >> >> > inactivity timer expired, stopping process 'buildassist'. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Shutdown >> >> > requested 'buildassist'. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Stopping >> >> > process 'buildassist'. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroying >> >> > interpreters. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroy >> >> > interpreter 'www.builderassist.com.au|'. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Cleanup >> >> > interpreter ''. >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Terminating >> >> > Python. >> >> > [Mon May 02 03:17:14 2011] [info] mod_wsgi (pid=22081): Python has >> >> > shutdown. >> >> > [Mon May 02 09:13:27 2011] [error] [client 120.156.100.6] Premature >> >> > end of script headers: django.wsgi >> >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature >> >> > end of script headers: django.wsgi >> >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature >> >> > end of script headers: django.wsgi >> >> > ---relaod apache here >> >> > [Mon May 02 10:21:26 2011] [info] mod_wsgi (pid=499): Attach >> >> > interpreter ''. >> >> > [Mon May 02 10:21:28 2011] [info] mod_wsgi (pid=499): Create >> >> > interpreter 'www.builderassist.com.au|'. >> >> > [Mon May 02 10:21:28 2011] [info] [client 120.156.100.6] mod_wsgi >> >> > (pid=499, process='buildassist', >> >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ >> >> > django/buildassist/django.wsgi'. >> >> >> > I am getting the same thing with the other sites, happens about 3 or 4 >> >> > times a week and requires apache reload to fix. >> >> >> > Any ideas? >> >> >> > Thanks, >> >> > Paddy >> >> >> > ----- virtual host >> >> > -------------------------------------------------------------- >> >> >> > WSGIDaemonProcess buildassist user=django group=django threads=25 >> >> > display-name=%{GROUP} inactivity-timeout=3600 >> >> > WSGIProcessGroup buildassist >> >> >> > WSGIScriptAlias / /var/django/buildassist/django.wsgi >> >> >> > ----- django.wsgi >> >> > --------------------------------------------------------------- >> >> >> > ALLDIRS = ['/usr/local/pythonenv/DJANGO_LEGACY_1.2/lib/python2.5/site- >> >> > packages/'] >> >> > APP_NAME = 'buildassist' >> >> > import os, sys, site >> >> >> > # virtualenv stuff ------------------------------ >> >> > prev_sys_path = list(sys.path) >> >> >> > for directory in ALLDIRS: >> >> > site.addsitedir(directory) >> >> >> > new_sys_path = [] >> >> >> > for item in list(sys.path): >> >> > if item not in prev_sys_path: >> >> > new_sys_path.append(item) >> >> > sys.path.remove(item) >> >> > sys.path[:0] = new_sys_path >> >> >> > # -------------------------------- >> >> >> > sys.path.append('/var/django') >> >> > sys.path.append('/var/django/%s' % APP_NAME) >> >> >> > os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % APP_NAME >> >> > os.environ['PYTHON_EGG_CACHE'] = '/tmp/eggs' >> >> >> > import django.core.handlers.wsgi >> >> >> > _application = django.core.handlers.wsgi.WSGIHandler() >> >> >> > import posixpath >> >> >> > def application(environ, start_response): >> >> > # Wrapper to set SCRIPT_NAME to actual mount point. >> >> > environ['SCRIPT_NAME'] = posixpath.dirname(environ['SCRIPT_NAME']) >> >> > if environ['SCRIPT_NAME'] == '/': >> >> > environ['SCRIPT_NAME'] = '' >> >> > return _application(environ, start_response) >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "modwsgi" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/modwsgi?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "modwsgi" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/modwsgi?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/modwsgi?hl=en. > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
