On Wed, Oct 07, 2009 at 05:21:00PM -0500, Ben Fyvie wrote: > Thanks Willy, > > > it's clear that the server unexpectedly closed the connection or died > > You are absolutely correct. We've found that the problem is not to do with > malformed headers, so we don't need to debug that like I had previously > thought we might.
OK (...) > After doing all of this we found that we were still getting 502 errors. One > of which was this: > > Oct 7 11:31:28 localhost haproxy[22110]: 127.0.0.1:36222 > [07/Oct/2009:11:31:27.522] nightingalenotes_staging > nightingalenotes_staging/nightingalenotes_5000 0/0/0/-1/861 502 204 - - SL-- > 0/0/0/0/0 0/0 "GET > /check_session?authenticity_token=fa175d0290443acf8ae9f2cb3d9104098facc3f7 > HTTP/1.0" (...) correct analysis below > So what our problem really comes down to is why doesn't mongrel quietly stop > receiving requests after monit issues the initial "kill". (FYI - it is our > understanding that calling "mongrel stop" also issues a "kill" command so > there is no nicer way to ask it to shut down) > > The thing that really confuses us is that prior to putting haproxy into > place we didn't receive any problems when a mongrel instance was restarted. > This leads us to believe that there must be something that haproxy is doing > that prevents the mongrel instance from shutting down nicely when issued the > "kill" command. Does haproxy keep a constant connection with the server > (mongrel in our case)??? No, but one thing is possible : haproxy regulates traffic between servers and ensures the queue is served as fast as possible. This means that as soon as one connection is released, another pending one is immediately served, just a few microseconds later. It is fairly possible that mongrel needs to have some idle time between requests to decide to soft-stop. This would be stupid, but everything is possible with this server... The correct way of performing a soft stop is to stop listening first, even if there's a request being processed. But clearly I'm not sure mongrel does it right. Another thing to investigate : why on earth is it required to constantly kill that server ??? Normally this should only happen in extreme cases, may be a few times a year or even a few times a month. In these situations, you wouldn't mind that much about one or two 502 responses being sent. Also another thing : since this server is limited to only one connection, as long as a request is being served, an HTTP health check cannot be served. This means you have to set a large enough "fall" parameter for your checks in order not to detect a server as down while it's processing a request. Of course this prevents haproxy from quickly detecting a soft stop condition. Regards, Willy
