On 15 March 2013 21:22, Bartosz Wiklak <[email protected]> wrote: > Hi, I did all the tweaks but today the service stopped running again > (after 5 days). > > apache log: > [Fri Mar 15 10:02:33 2013] [error] [client ip] (70007)The timeout > specified has expired: mod_wsgi (pid=22530): Unable to get bucket brigade > for request. > [Fri Mar 15 10:03:00 2013] [error] [client ip] Script timed out before > returning headers: django-app.wsgi > [...] > [Fri Mar 15 10:39:44 2013] [error] server is within MinSpareThreads of > MaxClients, consider raising the MaxClients setting > > I've read the previous threads, I'll try to use inactivity-timeout=120. > As far as I understand, some threads are deadlocking, can I observe this > in i.e. "ps -aux" or logs? > In the apache logs I'm getting depreciation warnings from django (btw. how > can I shut it down?) but no errors, if I'll turn on stacktraces will I see > something interesting/new? >
It may not be possible to do it on your production system, but gdb can be used to determine the state of the process if it is deadlocking. http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB I was adding to mod_wsgi 4.0 some new functionality to check for blocked requests and dump stack traces on a forced shutdown due to blocking, but been so long since I was working on that, I don't even remember how far I got with it. I hope to go back and work out where I was up to with mod_wsgi 4.0 soon and restart work on it. Graham > Thanks for help, > Bartek > > > On Saturday, March 9, 2013 2:47:33 AM UTC+1, Graham Dumpleton wrote: > >> >> >> >> On 8 March 2013 17:12, Bartosz Wiklak <[email protected]> wrote: >> >>> Hello, >>> >>> I'm running Django+Apache+mod_wsgi: >>> >>> Apache/2.2.16 (Debian) mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 >>> OpenSSL/0.9.8o mod_wsgi/3.3 >>> >>> >> Are you using mod_python still? >> >> If you aren't, stop loading it into Apache. >> >> >>> Server MPM: Worker >>> threaded: yes (fixed thread count) >>> forked: yes (variable process count) >>> >>> >>> apache conf: >>> >>> WSGIScriptAlias / /home/ja/apps/django/django-**app.wsgi >>> WSGIProcessGroup app-site-ssl >>> >> >> Add the line: >> >> WSGIApplicationGroup %{GLOBAL} >> >> and see if he problem goes away. >> >> >>> WSGIDaemonProcess app-site-ssl processes=2 threads=16 >>> maximum-requests=1000 display-name=apache-app-wsgi-**ssl >>> XSendFile on >>> >>> wsgi sctipt: >>> >>> import newrelic.agent >>> newrelic.agent.initialize('/**home/ja/apps/newrelic.ini', 'production') >>> >>> >> If you look at 'Reports -> Capacity' in New Relic you will see under the >> App instance busy/analysis charts that it was sudden lock up. It was not >> caused by a gradual exhaustion of threads due to threads blocking on >> backend services. >> >> The sudden lock is suggestive of a Python deadlock. This is often caused >> when using a C extension module with Python where the C extension is used >> in a sub interpreter, which is the default under mod_wsgi. >> >> The setting of: >> >> WSGIApplicationGroup %{GLOBAL} >> >> will force use of the main interpreter and avoid problems with such C >> extensions. For more details read: >> >> http://code.google.com/p/**modwsgi/wiki/**ApplicationIssues#Python_** >> Simplified_GIL_State_API<http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API> >> >> It would also be advisable not to load mod_python if you no longer need >> it. >> >> And no I am not psychic in knowing that chart in New Relic shows that. I >> work there are wrote the Python agent. :-) >> >> Hope you don't mind me sneaking a look. >> >> >> >>> import sys >>> import os >>> import os.path >>> >>> sys.path.append(os.path.**dirname(__file__)) >>> os.environ['DJANGO_SETTINGS_**MODULE'] = 'app.settings' >>> >>> from django.core.handlers.wsgi import WSGIHandler >>> application = WSGIHandler() >>> >>> application = newrelic.agent.wsgi_**application()(application) >>> >>> >> For Django you don't need this line as instrumentation automatically >> wraps django.core.handlers.**wsgi:WSGIHandler instance. >> >> Graham >> >> >>> >>> >>> Once a day my site stops responding. I restart apache and everything >>> backs to normal, today the server start respodning after 3 hours without >>> manual restart. >>> In apache error logs I'm getting: >>> >>> >>> [Fri Mar 08 20:07:44 2013] [error] [client ip] Script timed out before >>> returning headers: django-app.wsgi >>> [..,] lost of it >>> [Fri Mar 08 20:50:46 2013] [error] server is within MinSpareThreads of >>> MaxClients, consider raising the MaxClients setting >>> [Fri Mar 08 20:56:45 2013] [error] server reached MaxClients setting, >>> consider raising the MaxClients setting >>> [Fri Mar 08 22:49:48 2013] [warn] child process 913 still did not exit, >>> sending a SIGTERM >>> [...] several lines for different pids >>> [Fri Mar 08 22:49:49 2013] [error] [client 204.93.223.151] >>> (4)Interrupted system call: mod_wsgi (pid=14883): Unable to connect to WSGI >>> daemon process >>> 'app-site-ssl' on '/var/run/apache2/wsgi.14347.**0.1.sock' after >>> multiple attempts. >>> [...] >>> [Fri Mar 08 22:49:50 2013] [warn] child process 913 still did not exit, >>> sending a SIGTERM >>> [...] >>> [Fri Mar 08 22:49:55 2013] [notice] caught SIGTERM, shutting down >>> [Fri Mar 08 22:49:59 2013] [error] python_init: Python version mismatch, >>> expected '2.6.5+', found '2.6.6'. >>> [Fri Mar 08 22:49:59 2013] [error] python_init: Python executable found >>> '/usr/bin/python'. >>> [Fri Mar 08 22:49:59 2013] [error] python_init: Python path being used >>> '/usr/lib/python2.6/:/usr/lib/**python2.6/plat-linux2:/usr/** >>> lib/python2.6/lib-tk:/ >>> usr/lib/python2.6/lib-old:/**usr/lib/python2.6/lib-dynload'**. >>> [Fri Mar 08 22:49:59 2013] [notice] mod_python: Creating 8 session >>> mutexes based on 6 max processes and 25 max threads. >>> [Fri Mar 08 22:49:59 2013] [notice] mod_python: using mutex_directory >>> /tmp >>> [Fri Mar 08 22:49:59 2013] [notice] Apache/2.2.16 (Debian) >>> mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_wsgi/3.3 >>> configured -- resuming normal operations >>> Service is back again >>> >>> What can I do? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to modwsgi+u...@**googlegroups.com. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at >>> http://groups.google.com/**group/modwsgi?hl=en<http://groups.google.com/group/modwsgi?hl=en> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
