2010/1/28 Alec Flett <[email protected]>:
> We have a mod_wsgi 3.0c5 instance (just upgraded to 3.1 yesterday, but
> haven't deployed the apache install with that version yet) that seems to
> have this problem where some of the daemons don't restart when they've lived
> their active lifetime. So what happens is the apache runs along happily for
> days at a time, and then all the sudden (over the course of a few hours) all
> the requests get slow, and the number of occupied apache children (i.e.
> apache slots, not WSGI daemons) goes way up. When I log into the machine,
> there are only a few (in one example, only 7 of the original 48 that we
> started with) WSGI daemons hanging on for dear life, handling as many
> requests as they can.
>
> At the same time, we recently discovered (like a month ago) that we had a C
> extension (jsonlib2) that was crashing the python interpreter because of a
> refcounting bug. (We have since fixed the bug in jsonlib2, if anyone here is
> using it!)
>
> what I'm wondering is, if the WSGI daemon crashed hard (i.e. Bus Error, etc)
> what is the expected behavior of mod_wsgi? It looks like mod_wsgi should
> handle a SIGINT sent to a child, and restart the daemon, but what about a
> crash?
Should restart on a crash automatically.
One cause of what you are seeing is Python threads being deadlocked
and over time causing available threads to be used up.
Are you using multithread daemons? Is your code and third party
modules thread safe?
Try setting 'inactivity-timeout=120' as option to WSGIDaemonProcess.
This will cause a process to be automatically restarted if process
idle for 2 minutes, but will also force a restart of process where
there are active requests, but none of them have read any request body
or written and response content in 2 minutes. This can therefore be
used as a fail safe for processes that get stuck due to all threads
deadlocking.
I would also suggest setting LogLevel to 'info' so that additional
information printed out in error logs about process restarts.
As to finding out if Python threads stuck, modify your program to have
a URL which does the following:
import sys
import traceback
def stacktraces():
code = []
for threadId, stack in sys._current_frames().items():
code.append("\n# ThreadID: %s" % threadId)
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename,
lineno, name))
if line:
code.append(" %s" % (line.strip()))
return code
def application(environ, start_response):
status = '200 OK'
output = '\n'.join(stacktraces())
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
and either log the results or send it as response.
This way you might get an idea what request threads are actually doing.
Let me know what you find and also post your actual daemon mode configuration.
Graham
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en.
