Ok, dmirr is down again... You are going to need to look in to this as I'm not sure how to even debug a python webapp like this:
[Mon Jun 20 07:46:55 2011] [error] app = CacheMiddleware(app, config) [Mon Jun 20 07:54:58 2011] [warn] [client 173.203.5.222] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 07:54:58 2011] [error] [client 173.203.5.222] Premature end of script headers: health_check.py [Mon Jun 20 07:56:58 2011] [warn] [client 173.203.5.222] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 07:57:31 2011] [warn] [client 64.39.5.123] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 07:57:31 2011] [error] [client 64.39.5.123] Premature end of script headers: health_check.py [Mon Jun 20 07:59:31 2011] [warn] [client 64.39.5.123] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 08:11:54 2011] [warn] [client 173.203.5.222] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 08:11:54 2011] [error] [client 173.203.5.222] Premature end of script headers: health_check.py [Mon Jun 20 08:13:34 2011] [error] [client 8.20.213.37] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:13:38 2011] [error] [client 8.20.213.27] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:13:41 2011] [error] [client 8.19.240.68] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:13:52 2011] [error] [client 200.131.9.130] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:13:54 2011] [warn] [client 173.203.5.222] Timeout waiting for output from CGI script /var/www/scripts/health_check.py [Mon Jun 20 08:13:55 2011] [error] [client 91.121.180.211] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:15:28 2011] [error] [client 79.143.180.49] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:15:46 2011] [error] [client 209.134.48.153] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:15:53 2011] [error] [client 203.199.72.210] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:15:57 2011] [error] [client 128.32.32.220] Premature end of script headers: dmirr.wsgi [Mon Jun 20 08:16:06 2011] [error] [client 92.247.12.210] Premature end of script headers: dmirr.wsgi --- Jeffrey Ness Linux System Engineer IT Operations [Development] Rackspace Hosting & IUS Community On Jun 20, 2011, at 7:48 AM, Jeff Ness wrote: > Another reload and the app seems to be back up. > > I'm going to keep the error_log parsing in the background to see if it > happens again. > > --- > Jeffrey Ness > Linux System Engineer > IT Operations [Development] > Rackspace Hosting & IUS Community > > On Jun 20, 2011, at 7:45 AM, Jeff Ness wrote: > >> I don't think it has anything to do with logrotate as its happened again.... >> >> [root@322729-dmirr01 ~]# tail /var/log/httpd/error_log >> [Mon Jun 20 07:43:19 2011] [error] [client 209.17.170.226] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:43:33 2011] [error] [client 128.32.62.158] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:43:41 2011] [error] [client 72.3.236.90] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:43:44 2011] [error] [client 60.53.114.7] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:43:57 2011] [error] [client 193.191.154.73] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:43:58 2011] [error] [client 121.15.87.66] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:44:17 2011] [error] [client 137.118.0.254] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:44:19 2011] [error] [client 67.23.38.216] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:44:26 2011] [error] [client 128.61.143.100] Premature end of >> script headers: dmirr.wsgi >> [Mon Jun 20 07:44:28 2011] [error] [client 85.214.44.166] Premature end of >> script headers: dmirr.wsgi >> >> >> --- >> Jeffrey Ness >> Linux System Engineer >> IT Operations [Development] >> Rackspace Hosting & IUS Community >> >> On Jun 20, 2011, at 7:26 AM, Bj Dierkes wrote: >> >>> Sounds like maybe something to do with logrotate running on Sunday's.... >>> Hmm.... Once MF is beta I'm going to revisit dMirr. >>> >>> Sent from my iPhone >>> >>> On Jun 20, 2011, at 7:12 AM, "Jeff Ness" <[email protected]> wrote: >>> >>>> Seems dmirr had issues again, this time not related to full disk space: >>>> >>>> [Sun Jun 19 22:59:12 2011] [error] obj = object.__new__(cls, *args, **kw) >>>> [Sun Jun 19 22:59:13 2011] [error] >>>> /var/www/vhosts/dmirr.iuscommunity.org/dmirr-env/lib64/python2.6/site-packages/TurboGears2-2.0.3-py2.6. >>>> egg/tg/configuration.py:501: DeprecationWarning: Cache options should >>>> start with cache. instead of cache_ >>>> [Sun Jun 19 22:59:13 2011] [error] app = CacheMiddleware(app, config) >>>> [Sun Jun 19 23:57:07 2011] [error] [client 66.249.72.122] Error - <class >>>> 'repoze.what.predicates.NotAuthorizedError'>: >>>> [Mon Jun 20 01:15:42 2011] [error] [client 66.249.72.122] Error - <class >>>> 'repoze.what.predicates.NotAuthorizedError'>: >>>> [Mon Jun 20 01:28:09 2011] [error] [client 213.129.75.1] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:12 2011] [error] [client 163.43.163.48] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:13 2011] [error] [client 72.72.203.106] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:25 2011] [error] [client 193.110.195.10] Premature end >>>> of script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:26 2011] [error] [client 146.115.38.39] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:43 2011] [error] [client 193.85.153.130] Premature end >>>> of script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:46 2011] [error] [client 61.197.227.129] Premature end >>>> of script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:47 2011] [error] [client 87.237.215.129] Premature end >>>> of script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:49 2011] [error] [client 134.99.64.239] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:57 2011] [error] [client 196.15.240.2] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:28:58 2011] [error] [client 174.143.56.162] Premature end >>>> of script headers: dmirr.wsgi >>>> [Mon Jun 20 01:29:03 2011] [error] [client 50.17.177.64] Premature end of >>>> script headers: dmirr.wsgi >>>> [Mon Jun 20 01:29:03 2011] [error] [client 69.20.65.82] Premature end of >>>> script headers: dmirr.wsgi >>>> >>>> --- >>>> >>>> I issued a httpd restart and dmirr went back to normal. >>>> >>>> --- >>>> Jeffrey Ness >>>> Linux System Engineer >>>> IT Operations [Development] >>>> Rackspace Hosting & IUS Community >>>> >>>> On Jun 20, 2011, at 2:04 AM, <[email protected]> >>>> <[email protected]> wrote: >>>> >>>>> See <http://jenkins.iuscommunity.org/job/verify-mirrorlist/3104/> >>>>> >>>>> ------------------------------------------ >>>>> Started by timer >>>>> Building on master >>>>> [workspace] $ /bin/sh -xe /tmp/hudson200383432980641728.sh >>>>> + /usr/bin/curl -s >>>>> 'http://dmirr.iuscommunity.org/mirrorlist?repo=ius-el4&arch=i386' >>>>> + grep ius/stable >>>>> + exit 1 >>>>> >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~ius-coredev >>>>> Post to : [email protected] >>>>> Unsubscribe : https://launchpad.net/~ius-coredev >>>>> More help : https://help.launchpad.net/ListHelp >>>> >> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~ius-coredev >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~ius-coredev >> More help : https://help.launchpad.net/ListHelp > > > _______________________________________________ > Mailing list: https://launchpad.net/~ius-coredev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~ius-coredev > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~ius-coredev Post to : [email protected] Unsubscribe : https://launchpad.net/~ius-coredev More help : https://help.launchpad.net/ListHelp
