Hi Emmanuel, Thanks for pointing me to that documentation, does help explain why it is implemented.
I do suspect it is down to Apache, but because the issue is related to connection closing, and HAProxy is always at the other end of that connection, I figured users on this list may have experienced similar issues. I am using Apache from the CentOS 5 repository (Apache 2.2.3), mpm prefork (StartServers: 200, MinSpareServers: 50, MaxSpareServers: 400, ServerLimit: 1024, MaxClients: 1024) Alex 2010/1/20 Emmanuel Bailleul <[email protected]> > Hi Alex, > > The 'lingering close' Apache uses is briefly described here : > http://httpd.apache.org/docs/2.1/misc/perf-tuning.html#compiletime > > Your issue seems to be rather with Apache, but could you tell us which > version you are using and how it is configured (mpm) ? > > Emmanuel > > De : Alex Forrow [mailto:[email protected]] > Envoyé : mardi 19 janvier 2010 23:12 > À : [email protected] > Objet : Apache CLOSING state > > Hi all, > > I've been using HAProxy for about a year on a high traffic website and can > say nothing but good things about it. Recently however I've run into an > issue that I'm hoping someone may be able to help with. > > After a recent increase in traffic, we have been unexpectedly maxing out > the 1000 connection limit (on HAProxy) to our Apache instances at our > busiest point of the day. Looking at the Apache mod_status page at this > time, it seems that many of the slots are in the 'CLOSING' state. We do > normally have some slots in this state, but it seems to increase suddenly at > about the same time each day, and decrease in the the same fashion a few > hours later. I believe it is only the connection limit on HAProxy that > prevents the situation becoming any worse (requests are queued/dropped). An > example graph of of slot usage is available at > http://sysadmin.favsys.net/pub/slots.png > > The Apache 'CLOSING' state seems poorly documented, but after a quick look > at the source, it seems to be a some internal linger code to avoid closing > connections before the last data packet has been received by the browser ( > http://svn.apache.org/viewvc/httpd/httpd/trunk/server/connection.c?view=markup > ). > > We have HAProxy running on two identical machines on the same subnet, and > sometimes I am able to alleviate the problem by switching the virtual IP > when this problem occurs. However on our most recent traffic spike, this > hasn't been helping for any reasonable amount of time. After an initial > spike, HAProxy calms down for a minute or so, then returns to the previous > maxed-out state. > > Here are some details about the setup: > . HAProxy 1.3.22 > . Roundrobin balancing to 4 Apache nodes > . Using 'option httpclose' > . Keep-Alive disabled on all Apache servers (though I have tried enabling > it on a single server for investigation purposes) > I have recently tried adding another Apache instance to the pool, and this > doesn't seem to help. > > Any help or guidance would be greatly appreciated. Let me know if there is > any other information I can provide. > > > Regards, > > Alex > >
