Hello Geoff, On Wed, Oct 13, 2010 at 11:45:02AM -0500, glovett wrote: > Hello all, I need some advice on how to tune haproxy to react more quickly > when an http service goes down. Recently, haproxy emitted 503s to clients > for 29 seconds when one of three servers in a pool went down. "Haproxy may > emit the following status codes by itself... 503 when no server was > available to handle the request, or in response to monitoring requests which > match the 'monitor fail' condition." However there were other servers > available. Here's the timeline: > > 13:47:58 - http service offline > 13:48:00 - 13:48:29 - 503 returned to clients of offline server, other > servers in pool continued to service clients > 13:48:30-13:48:34 - haproxy stopped dispatching client requests to offline > server > 13:48:34 - offline server marked down in haproxy log > > Here's an example line from the 503 errors: > Oct 5 13:48:00 127.0.0.1 haproxy[20308]: 1.1.1.4:65054 > [05/Oct/2010:13:48:00.662] web_pool/web4 0/0/0/0/1 503 513 - - ---- 8/1/1/0/0 > 0/0 "GET / HTTP/1.0"
>From what I see in the timers and flags, it's not haproxy that returned the 503, but the server, and haproxy forwarded it. If the the connection had failed, then you'd have flags such as "SC" indicating a failed connection, and the connection timer would have been marked "-1". > Here's what haproxy had to say when it marked web4 offline: > Oct 5 13:48:34 127.0.0.1 haproxy[20308]: Server web_pool/web4 is DOWN. 2 > active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining > in queue. The fact that haproxy detected the server down a little time after the 503 also seems to suggest that the server was becoming ill and has emitted the 503 itself. BTW, your config is quite clean, there is nothing in it which could cause a 503 to appear on a single down server (retries & redispatch are here, no persistence, no queue). Hoping this helps, Willy
