Many thanks for the feedback. Emmanuel
De : Alex Forrow [mailto:[email protected]] Envoyé : lundi 1 février 2010 17:26 À : Emmanuel Bailleul Cc : [email protected] Objet : Re: Apache CLOSING state Just to provide an update to this issue, the problem was caused by mis-configured conntrack. Temporary removal of this module solved the issue. On 22 January 2010 14:46, Alex Forrow <[email protected]<mailto:[email protected]>> wrote: Hi Emmanuel, Thanks for pointing me to that documentation, does help explain why it is implemented. I do suspect it is down to Apache, but because the issue is related to connection closing, and HAProxy is always at the other end of that connection, I figured users on this list may have experienced similar issues. I am using Apache from the CentOS 5 repository (Apache 2.2.3), mpm prefork (StartServers: 200, MinSpareServers: 50, MaxSpareServers: 400, ServerLimit: 1024, MaxClients: 1024) Alex 2010/1/20 Emmanuel Bailleul <[email protected]<mailto:[email protected]>> Hi Alex, The 'lingering close' Apache uses is briefly described here : http://httpd.apache.org/docs/2.1/misc/perf-tuning.html#compiletime Your issue seems to be rather with Apache, but could you tell us which version you are using and how it is configured (mpm) ? Emmanuel De : Alex Forrow [mailto:[email protected]<mailto:[email protected]>] Envoyé : mardi 19 janvier 2010 23:12 À : [email protected]<mailto:[email protected]> Objet : Apache CLOSING state Hi all, I've been using HAProxy for about a year on a high traffic website and can say nothing but good things about it. Recently however I've run into an issue that I'm hoping someone may be able to help with. After a recent increase in traffic, we have been unexpectedly maxing out the 1000 connection limit (on HAProxy) to our Apache instances at our busiest point of the day. Looking at the Apache mod_status page at this time, it seems that many of the slots are in the 'CLOSING' state. We do normally have some slots in this state, but it seems to increase suddenly at about the same time each day, and decrease in the the same fashion a few hours later. I believe it is only the connection limit on HAProxy that prevents the situation becoming any worse (requests are queued/dropped). An example graph of of slot usage is available at http://sysadmin.favsys.net/pub/slots.png The Apache 'CLOSING' state seems poorly documented, but after a quick look at the source, it seems to be a some internal linger code to avoid closing connections before the last data packet has been received by the browser (http://svn.apache.org/viewvc/httpd/httpd/trunk/server/connection.c?view=markup). We have HAProxy running on two identical machines on the same subnet, and sometimes I am able to alleviate the problem by switching the virtual IP when this problem occurs. However on our most recent traffic spike, this hasn't been helping for any reasonable amount of time. After an initial spike, HAProxy calms down for a minute or so, then returns to the previous maxed-out state. Here are some details about the setup: . HAProxy 1.3.22 . Roundrobin balancing to 4 Apache nodes . Using 'option httpclose' . Keep-Alive disabled on all Apache servers (though I have tried enabling it on a single server for investigation purposes) I have recently tried adding another Apache instance to the pool, and this doesn't seem to help. Any help or guidance would be greatly appreciated. Let me know if there is any other information I can provide. Regards, Alex
