Thanks Graham I don't really need it so will remove and see how I go. Paddy
On May 2, 2:17 pm, Graham Dumpleton <[email protected]> wrote: > Unless you have a real need for it, get rid of the inactivity-timeout > option. This will avoid daemon processes restarting and so potentially > triggering the issue. > > On 2 May 2011 14:10, Paddy Joy <[email protected]> wrote: > > > Thanks Graham I will report back anything I can find. In my situation > > every time the "premature end of script" occurs the daemon process is > > not recreated which is a headache with sites dropping offline. > > > Paddy > > > On May 2, 1:25 pm, Graham Dumpleton <[email protected]> > > wrote: > >> The failure to create replacement daemon mode processes is something > >> that has been seen, but vary rarely, with prefork MPM. It has been an > >> issue that has been hard to track down as no useful information to > >> really say what is preceding that. For example, whether there is a > >> temporary resource issue on the host for example. Right now there is > >> no indication that it is a mod_wsgi specific issue. So, if you do > >> manage to capture any evidence of threads being stuck at a particular > >> place, it may be useful. > > >> Graham > > >> On 2 May 2011 12:48, Paddy Joy <[email protected]> wrote: > > >> > Graham, > > >> > Thanks for your prompt help as usual. I have installed the code in one > >> > of the applications so hopefully I can catch the timeout. It's strange > >> > because it is different urls on different sites that seems to start > >> > the issue, there seems to be no pattern. Some of the code on the older > >> > sites hasn't been touched in months. > > >> > When the daemon process reaches the timeout and is shutdown, shouldn't > >> > a new process be started the next time a request comes in? I need to > >> > put something in place for the short term so that I can reload apache > >> > once this situation occurs, should I use a simple script to monitor if > >> > each application still has a process running or what would you > >> > recommend? > > >> > Paddy > > >> > On May 2, 11:19 am, Graham Dumpleton <[email protected]> > >> > wrote: > >> >> First impressions would suggest that request handler threads are > >> >> blocking on some operation and simply never completing. Eventually > >> >> what happens is that there is a period long enough that the inactivity > >> >> timeout is kicking in and the daemon process is shutdown. When this > >> >> occurs the connection from Apache child process for the blocked > >> >> request is broken and the message 'Premature end of script headers: > >> >> django.wsgi' results. > > >> >> I would suggest integrating some variant of: > > >> >>http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_... > > >> >> into your application such that you have a means of triggering a dump > >> >> of what all request handler threads are doing. By luck you may be able > >> >> to trigger this when there are some of these blocked threads and work > >> >> out what they are blocking on. Since your inactivity timeout is an > >> >> hour long, then that may not be too hard actually. Set up a cron job > >> >> to trigger the stack dump every 20 minutes and see if you see request > >> >> threads that are always stuck at same point. Alternative to cron is > >> >> just modify that example to do a sleep() within the background thread > >> >> itself and just automatically dump out every 20 minutes. > > >> >> Graham > > >> >> On 2 May 2011 11:08, Paddy Joy <[email protected]> wrote: > > >> >> > Hi, > > >> >> > Hope someone can help debug the following issue I am having. > > >> >> > I have been running approx 15 django sites with mod_wsgi for a few > >> >> > years now with no problems. In the last month I am getting sites that > >> >> > randomly stop working, the process is gone and apache returns 500 > >> >> > status. I have set logging level to info but it still doesn't give me > >> >> > enough info to pinpoint the problem. The only notable change I made to > >> >> > the server was a switch to apache mpm-itk a couple of months back. > > >> >> > Server config is as follows, example of django.wsgi and virtual host > >> >> > at bottom of email. > >> >> > apache 2.2.8 mpm-itk > >> >> > python 2.5.2 > >> >> > mod_wsgi 3.3 daemon mode > >> >> > mod_python not installed > > >> >> > Example: > > >> >> > apache access log, site ok at 2am when googlebot visited then 500 > >> >> > error in morning > >> >> > 66.249.71.166 - - [02/May/2011:02:17:11 +1000] "GET /password_reset/ > >> >> > HTTP/1.1" 200 6071 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; > >> >> > +http://www.google.com/bot.html)" > >> >> > 120.156.100.6 - - [02/May/2011:09:08:27 +1000] "GET /favicon.ico HTTP/ > >> >> > 1.1" 500 739 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) > >> >> > Gecko/20100914 Firefox/3.6.10" > > >> >> > Error log: > >> >> > [Mon May 02 02:17:11 2011] [info] mod_wsgi (pid=22081): Create > >> >> > interpreter 'www.builderassist.com.au|'. > >> >> > [Mon May 02 02:17:11 2011] [info] [client 66.249.71.166] mod_wsgi > >> >> > (pid=22081, process='buildassist', > >> >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > >> >> > django/buildassist/django.wsgi'. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Daemon process > >> >> > inactivity timer expired, stopping process 'buildassist'. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Shutdown > >> >> > requested 'buildassist'. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Stopping > >> >> > process 'buildassist'. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroying > >> >> > interpreters. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Destroy > >> >> > interpreter 'www.builderassist.com.au|'. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Cleanup > >> >> > interpreter ''. > >> >> > [Mon May 02 03:17:13 2011] [info] mod_wsgi (pid=22081): Terminating > >> >> > Python. > >> >> > [Mon May 02 03:17:14 2011] [info] mod_wsgi (pid=22081): Python has > >> >> > shutdown. > >> >> > [Mon May 02 09:13:27 2011] [error] [client 120.156.100.6] Premature > >> >> > end of script headers: django.wsgi > >> >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > >> >> > end of script headers: django.wsgi > >> >> > [Mon May 02 10:21:24 2011] [error] [client 120.156.100.6] Premature > >> >> > end of script headers: django.wsgi > >> >> > ---relaod apache here > >> >> > [Mon May 02 10:21:26 2011] [info] mod_wsgi (pid=499): Attach > >> >> > interpreter ''. > >> >> > [Mon May 02 10:21:28 2011] [info] mod_wsgi (pid=499): Create > >> >> > interpreter 'www.builderassist.com.au|'. > >> >> > [Mon May 02 10:21:28 2011] [info] [client 120.156.100.6] mod_wsgi > >> >> > (pid=499, process='buildassist', > >> >> > application='www.builderassist.com.au|'): Loading WSGI script '/var/ > >> >> > django/buildassist/django.wsgi'. > > >> >> > I am getting the same thing with the other sites, happens about 3 or 4 > >> >> > times a week and requires apache reload to fix. > > >> >> > Any ideas? > > >> >> > Thanks, > >> >> > Paddy > > >> >> > ----- virtual host > >> >> > -------------------------------------------------------------- > > >> >> > WSGIDaemonProcess buildassist user=django group=django threads=25 > >> >> > display-name=%{GROUP} inactivity-timeout=3600 > >> >> > WSGIProcessGroup buildassist > > >> >> > WSGIScriptAlias / /var/django/buildassist/django.wsgi > > >> >> > ----- django.wsgi > >> >> > --------------------------------------------------------------- > > >> >> > ALLDIRS = ['/usr/local/pythonenv/DJANGO_LEGACY_1.2/lib/python2.5/site- > >> >> > packages/'] > >> >> > APP_NAME = 'buildassist' > >> >> > import os, sys, site > > >> >> > # virtualenv stuff ------------------------------ > >> >> > prev_sys_path = list(sys.path) > > >> >> > for directory in ALLDIRS: > >> >> > site.addsitedir(directory) > > >> >> > new_sys_path = [] > > >> >> > for item in list(sys.path): > >> >> > if item not in prev_sys_path: > >> >> > new_sys_path.append(item) > >> >> > sys.path.remove(item) > >> >> > sys.path[:0] = new_sys_path > > >> >> > # -------------------------------- > > >> >> > sys.path.append('/var/django') > >> >> > sys.path.append('/var/django/%s' % APP_NAME) > > >> >> > os.environ['DJANGO_SETTINGS_MODULE'] = '%s.settings' % APP_NAME > >> >> > os.environ['PYTHON_EGG_CACHE'] = '/tmp/eggs' > > >> >> > import django.core.handlers.wsgi > > >> >> > _application = django.core.handlers.wsgi.WSGIHandler() > > >> >> > import posixpath > > >> >> > def application(environ, start_response): > >> >> > # Wrapper to set SCRIPT_NAME to actual mount point. > >> >> > environ['SCRIPT_NAME'] = posixpath.dirname(environ['SCRIPT_NAME']) > >> >> > if environ['SCRIPT_NAME'] == '/': > >> >> > environ['SCRIPT_NAME'] = '' > >> >> > return _application(environ, start_response) > > >> >> > -- > >> >> > You received this message because you are subscribed to the Google > >> >> > Groups "modwsgi" group. > >> >> > To post to this group, send email to [email protected]. > >> >> > To unsubscribe from this group, send email to > >> >> > [email protected]. > >> >> > For more options, visit this group > >> >> > athttp://groups.google.com/group/modwsgi?hl=en. > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "modwsgi" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/modwsgi?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/modwsgi?hl=en. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
