Hi, On Mon, Oct 11, 2010 at 02:29:50PM -0700, [email protected] wrote: > When watching the stats page for haproxy, I'm seeing, on a > semi-regular basis, web hosts changing to yellow (Active up, going > down) then back to green again on the next refresh. I don't ever see > any of the hosts going to any other state, only between yellow and > green. We did see this a lot when we started using haproxy, but I > discovered that that was because of the iptables connection queue > filling up. Currently, iptables is not running on either the load > balancer or the web back-ends. > > Is there any obvious reason for this? Is it something we should be > concerned about? If I refresh the stats page every few seconds I see > this happen maybe 2 - 3 times a minute, occasionally more.
During startup, it's normal, because haproxy does not yet know what state the servers are in. So it assumes that they're almost dead so that one health check is enough to determine their initial state. If you see that later, most often this is caused by too short check intervals for your servers. For instance, if your servers sometimes take 1.5 seconds to respond to a health check and you check every second, then such late responses will be counted as failures. What you observe can also happen with mongrel-based servers. Mongrel has a very "funny" architecture that makes it accept only one connection at a time. So while it is processing a request, it cannot respond to a check. If that's your case, what I'd really suggest is to slow down checks (eg: every 10 seconds with 3 retries), and that you make use of the "observe layer7" server options to use production traffic to detect server failures as soon as possible. Regards, Willy
