Hi,
I run a medium-sized website that uses mongrel/rails in the following
configuration:
-frontend reverse proxy to a cluster of 7 app servers
-each app server runs apache 2.2 with mod_proxy_balancer that balancers
the requests out into a mongrel cluster of 35 servers.
-each mongrel is version 1.0.1
the mongrel cluster config looks like:
port: "1620"
environment: production
address: 127.0.0.1
pid_file: log/mongrel.pid
servers: 35
The problem is, when the mongrel processes are sent a USR2 signal, some
of them seem to have threads running, that are waiting for data on a
socket that's been disconnected from the apache balancer long ago.
process 27457 has been sent a USR2 signal, and is waiting for its thread
to die:
[EMAIL PROTECTED] config]$ /usr/sbin/lsof | grep 27457
[snip]
ruby 27457 ilike 28u IPv4 214673805 TCP
*:49242 (LISTEN)
ruby 27457 ilike 29u IPv4 214673810 TCP
app04:49244->10.1.2.10:mysql (ESTABLISHED)
[snip]
Here's the strace of what it's actually doing:
[EMAIL PROTECTED] config]$ strace -p 27457
Process 27457 attached - interrupt to quit
select(29, [28], [], [], {0, 137000}) = 0 (Timeout)
gettimeofday({1171499150, 914277}, NULL) = 0
select(29, [28], [], [], {0, 175}) = 0 (Timeout)
gettimeofday({1171499150, 915224}, NULL) = 0
select(29, [28], [], [], {0, 0}) = 0 (Timeout)
time(NULL) = 1171499150
gettimeofday({1171499150, 915386}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=1017, ...}) = 0
write(2, "Wed Feb 14 16:25:50 PST 2007: Re"..., 86) = 86
write(2, "\n", 1) = 1
gettimeofday({1171499150, 915596}, NULL) = 0
write(2, "Waiting for 1 requests to finish"..., 58) = 58
Port 49242 is an irrelevant port - the mongrels listen on ports
1620-1655 (the mongrel in question has closed out its main port). It
seems that the mongrel is waiting for data on a port that has become
disconnected from the mod_proxy_balancer.
I could probably provide an end-to-end strace if needed, although
picking out the relevant bits could be tough.
-ben
_______________________________________________
Mongrel-users mailing list
Mongrel-users@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-users
