Hi all, I've been using HAProxy for about a year on a high traffic website and can say nothing but good things about it. Recently however I've run into an issue that I'm hoping someone may be able to help with.
After a recent increase in traffic, we have been unexpectedly maxing out the 1000 connection limit (on HAProxy) to our Apache instances at our busiest point of the day. Looking at the Apache mod_status page at this time, it seems that many of the slots are in the 'CLOSING' state. We do normally have some slots in this state, but it seems to increase suddenly at about the same time each day, and decrease in the the same fashion a few hours later. I believe it is only the connection limit on HAProxy that prevents the situation becoming any worse (requests are queued/dropped). An example graph of of slot usage is available at http://sysadmin.favsys.net/pub/slots.png The Apache 'CLOSING' state seems poorly documented, but after a quick look at the source, it seems to be a some internal linger code to avoid closing connections before the last data packet has been received by the browser ( http://svn.apache.org/viewvc/httpd/httpd/trunk/server/connection.c?view=markup ). We have HAProxy running on two identical machines on the same subnet, and sometimes I am able to alleviate the problem by switching the virtual IP when this problem occurs. However on our most recent traffic spike, this hasn't been helping for any reasonable amount of time. After an initial spike, HAProxy calms down for a minute or so, then returns to the previous maxed-out state. Here are some details about the setup: - HAProxy 1.3.22 - Roundrobin balancing to 4 Apache nodes - Using 'option httpclose' - Keep-Alive disabled on all Apache servers (though I have tried enabling it on a single server for investigation purposes) I have recently tried adding another Apache instance to the pool, and this doesn't seem to help. Any help or guidance would be greatly appreciated. Let me know if there is any other information I can provide. Regards, Alex
