Just to provide an update to this issue, the problem was caused by mis-configured conntrack. Temporary removal of this module solved the issue.
On 22 January 2010 14:46, Alex Forrow <[email protected]> wrote: > Hi Emmanuel, > > Thanks for pointing me to that documentation, does help explain why it is > implemented. > > I do suspect it is down to Apache, but because the issue is related to > connection closing, and HAProxy is always at the other end of that > connection, I figured users on this list may have experienced similar > issues. > > I am using Apache from the CentOS 5 repository (Apache 2.2.3), mpm prefork > (StartServers: 200, MinSpareServers: 50, MaxSpareServers: 400, ServerLimit: > 1024, MaxClients: 1024) > > > Alex > > 2010/1/20 Emmanuel Bailleul <[email protected]> > > Hi Alex, >> >> The 'lingering close' Apache uses is briefly described here : >> http://httpd.apache.org/docs/2.1/misc/perf-tuning.html#compiletime >> >> Your issue seems to be rather with Apache, but could you tell us which >> version you are using and how it is configured (mpm) ? >> >> Emmanuel >> >> De : Alex Forrow [mailto:[email protected]] >> Envoyé : mardi 19 janvier 2010 23:12 >> À : [email protected] >> Objet : Apache CLOSING state >> >> Hi all, >> >> I've been using HAProxy for about a year on a high traffic website and can >> say nothing but good things about it. Recently however I've run into an >> issue that I'm hoping someone may be able to help with. >> >> After a recent increase in traffic, we have been unexpectedly maxing out >> the 1000 connection limit (on HAProxy) to our Apache instances at our >> busiest point of the day. Looking at the Apache mod_status page at this >> time, it seems that many of the slots are in the 'CLOSING' state. We do >> normally have some slots in this state, but it seems to increase suddenly at >> about the same time each day, and decrease in the the same fashion a few >> hours later. I believe it is only the connection limit on HAProxy that >> prevents the situation becoming any worse (requests are queued/dropped). An >> example graph of of slot usage is available at >> http://sysadmin.favsys.net/pub/slots.png >> >> The Apache 'CLOSING' state seems poorly documented, but after a quick look >> at the source, it seems to be a some internal linger code to avoid closing >> connections before the last data packet has been received by the browser ( >> http://svn.apache.org/viewvc/httpd/httpd/trunk/server/connection.c?view=markup >> ). >> >> We have HAProxy running on two identical machines on the same subnet, and >> sometimes I am able to alleviate the problem by switching the virtual IP >> when this problem occurs. However on our most recent traffic spike, this >> hasn't been helping for any reasonable amount of time. After an initial >> spike, HAProxy calms down for a minute or so, then returns to the previous >> maxed-out state. >> >> Here are some details about the setup: >> . HAProxy 1.3.22 >> . Roundrobin balancing to 4 Apache nodes >> . Using 'option httpclose' >> . Keep-Alive disabled on all Apache servers (though I have tried enabling >> it on a single server for investigation purposes) >> I have recently tried adding another Apache instance to the pool, and this >> doesn't seem to help. >> >> Any help or guidance would be greatly appreciated. Let me know if there is >> any other information I can provide. >> >> >> Regards, >> >> Alex >> >> >
