I have a webserver running a Django application using mod_wsgi in
daemon mode, which seems to be experiencing intermittent problems with
apache processes hanging. The apache processes that are serving the
requests are the ones hanging (the wsgi daemons seem fine), which
eventually causes apache to reach MaxClients and block on new
requests.
The server is running Debian Lenny, with apache 2.2.9 installed from
packages, using the prefork MPM. mod_wsgi is at version 2.5, compiled
from source with no extra configure options. We also have nginx
running in front of apache serving most of the static content,
proxying the dynamic content to apache. WSGI is configured as follows:
WSGIDaemonProcess live user=www-live group=www-live processes=4
threads=25 maximum-requests=500
WSGIProcessGroup live
If I attach strace to the apache child processes, I can see that they
are blocked on a read() from fd #15. According to lsof, this fd is an
anonymous UNIX socket, which I assume is the connection from the
apache request-serving process to the wsgi daemon. Attaching strace to
the daemon process comes back with:
Process 20625 attached with 28 threads - interrupt to quit
[pid 20598] restart_syscall(<... resuming interrupted call ...>
<unfinished ...>
[pid 20601] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20600] select(0, NULL, NULL, NULL, {0, 384000} <unfinished ...>
[pid 20602] fcntl(12, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET,
start=0, len=0} <unfinished ...>
[pid 20599] select(0, NULL, NULL, NULL, {216, 388000} <unfinished ...>
[pid 20602] <... fcntl resumed> ) = 0
[pid 20602] futex(0x252c028, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 20601] <... futex resumed> ) = 0
[pid 20601] fcntl(12, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET,
start=0, len=0} <unfinished ...>
[pid 20602] <... futex resumed> ) = 1
[pid 20602] close(13) = 0
[pid 20602] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20603] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20604] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20605] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20606] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20607] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20608] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20609] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20610] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20611] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20612] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20613] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20614] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20615] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20616] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20617] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20618] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20619] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20620] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20621] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20622] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20623] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20624] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20625] futex(0x252c028, FUTEX_WAIT_PRIVATE, 2, NULL
<unfinished ...>
[pid 20600] <... select resumed> ) = 0 (Timeout)
[pid 20600] select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
[pid 20600] select(0, NULL, NULL, NULL, {1, 0}^C <unfinished ...>
This looks to be the same as the output from attaching to an idle
daemon process when running normally, which suggests to me that the
daemon processes are continuing to run as normal.
It seems to be quite a rare occurrence, most requests seem to go
through OK. Unfortunately, I have yet to find a way of reproducing the
issue - as far as I can see so far there's nothing specific that
triggers the hanging.
Does anyone have any ideas what could be causing this? Any suggestions
of what to look at when trying to narrow down the cause? (bear in mind
that my response time may vary, as I might have to wait for the
problem to arise again before I can test anything new)
--
Chris
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en
-~----------~----~----~----~------~----~------~--~---
