Hi, I did all the tweaks but today the service stopped running again (after 5 days).
apache log: [Fri Mar 15 10:02:33 2013] [error] [client ip] (70007)The timeout specified has expired: mod_wsgi (pid=22530): Unable to get bucket brigade for request. [Fri Mar 15 10:03:00 2013] [error] [client ip] Script timed out before returning headers: django-app.wsgi [...] [Fri Mar 15 10:39:44 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting I've read the previous threads, I'll try to use inactivity-timeout=120. As far as I understand, some threads are deadlocking, can I observe this in i.e. "ps -aux" or logs? In the apache logs I'm getting depreciation warnings from django (btw. how can I shut it down?) but no errors, if I'll turn on stacktraces will I see something interesting/new? Thanks for help, Bartek On Saturday, March 9, 2013 2:47:33 AM UTC+1, Graham Dumpleton wrote: > > > > > On 8 March 2013 17:12, Bartosz Wiklak <[email protected] <javascript:>>wrote: > >> Hello, >> >> I'm running Django+Apache+mod_wsgi: >> >> Apache/2.2.16 (Debian) mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 >> OpenSSL/0.9.8o mod_wsgi/3.3 >> >> > Are you using mod_python still? > > If you aren't, stop loading it into Apache. > > >> Server MPM: Worker >> threaded: yes (fixed thread count) >> forked: yes (variable process count) >> >> >> apache conf: >> >> WSGIScriptAlias / /home/ja/apps/django/django-app.wsgi >> WSGIProcessGroup app-site-ssl >> > > Add the line: > > WSGIApplicationGroup %{GLOBAL} > > and see if he problem goes away. > > >> WSGIDaemonProcess app-site-ssl processes=2 threads=16 >> maximum-requests=1000 display-name=apache-app-wsgi-ssl >> XSendFile on >> >> wsgi sctipt: >> >> import newrelic.agent >> newrelic.agent.initialize('/home/ja/apps/newrelic.ini', 'production') >> >> > If you look at 'Reports -> Capacity' in New Relic you will see under the > App instance busy/analysis charts that it was sudden lock up. It was not > caused by a gradual exhaustion of threads due to threads blocking on > backend services. > > The sudden lock is suggestive of a Python deadlock. This is often caused > when using a C extension module with Python where the C extension is used > in a sub interpreter, which is the default under mod_wsgi. > > The setting of: > > WSGIApplicationGroup %{GLOBAL} > > will force use of the main interpreter and avoid problems with such C > extensions. For more details read: > > > http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API > > It would also be advisable not to load mod_python if you no longer need it. > > And no I am not psychic in knowing that chart in New Relic shows that. I > work there are wrote the Python agent. :-) > > Hope you don't mind me sneaking a look. > > > >> import sys >> import os >> import os.path >> >> sys.path.append(os.path.dirname(__file__)) >> os.environ['DJANGO_SETTINGS_MODULE'] = 'app.settings' >> >> from django.core.handlers.wsgi import WSGIHandler >> application = WSGIHandler() >> >> application = newrelic.agent.wsgi_application()(application) >> >> > For Django you don't need this line as instrumentation automatically > wraps django.core.handlers.wsgi:WSGIHandler instance. > > Graham > > >> >> >> Once a day my site stops responding. I restart apache and everything >> backs to normal, today the server start respodning after 3 hours without >> manual restart. >> In apache error logs I'm getting: >> >> >> [Fri Mar 08 20:07:44 2013] [error] [client ip] Script timed out before >> returning headers: django-app.wsgi >> [..,] lost of it >> [Fri Mar 08 20:50:46 2013] [error] server is within MinSpareThreads of >> MaxClients, consider raising the MaxClients setting >> [Fri Mar 08 20:56:45 2013] [error] server reached MaxClients setting, >> consider raising the MaxClients setting >> [Fri Mar 08 22:49:48 2013] [warn] child process 913 still did not exit, >> sending a SIGTERM >> [...] several lines for different pids >> [Fri Mar 08 22:49:49 2013] [error] [client 204.93.223.151] (4)Interrupted >> system call: mod_wsgi (pid=14883): Unable to connect to WSGI daemon process >> 'app-site-ssl' on '/var/run/apache2/wsgi.14347.0.1.sock' after multiple >> attempts. >> [...] >> [Fri Mar 08 22:49:50 2013] [warn] child process 913 still did not exit, >> sending a SIGTERM >> [...] >> [Fri Mar 08 22:49:55 2013] [notice] caught SIGTERM, shutting down >> [Fri Mar 08 22:49:59 2013] [error] python_init: Python version mismatch, >> expected '2.6.5+', found '2.6.6'. >> [Fri Mar 08 22:49:59 2013] [error] python_init: Python executable found >> '/usr/bin/python'. >> [Fri Mar 08 22:49:59 2013] [error] python_init: Python path being used >> '/usr/lib/python2.6/:/usr/lib/python2.6/plat-linux2:/usr/lib/python2.6/lib-tk:/ >> usr/lib/python2.6/lib-old:/usr/lib/python2.6/lib-dynload'. >> [Fri Mar 08 22:49:59 2013] [notice] mod_python: Creating 8 session >> mutexes based on 6 max processes and 25 max threads. >> [Fri Mar 08 22:49:59 2013] [notice] mod_python: using mutex_directory /tmp >> [Fri Mar 08 22:49:59 2013] [notice] Apache/2.2.16 (Debian) >> mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_wsgi/3.3 >> configured -- resuming normal operations >> Service is back again >> >> What can I do? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected]<javascript:> >> . >> Visit this group at http://groups.google.com/group/modwsgi?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
