Hi Andy, On Thu, Feb 20, 2014 at 11:41:20AM -0600, Andy Walker wrote: > This is something that looks like it's come up a couple of times on the > mailing list: > http://thread.gmane.org/gmane.comp.web.haproxy/3253 > http://thread.gmane.org/gmane.comp.web.haproxy/6494 > > and maybe even a couple of times on StackExchange: > http://serverfault.com/questions/291467/haproxy-badreq-errors > http://serverfault.com/questions/285850/haproxy-display-a-badreq-badreqs-by-the-thousands > > Basically, we're having the same problem, but I think I may have come up > with the cause -- at least in our situation. It all began as sporadic > reports of users getting "The website is too busy to show the webpage -- > HTTP 408/HTTP 409" errors. The common thread seemed to be that it was > always IE users. We weren't seeing anything in the logs (because we had > *option > dontlognull* turned on), and we were unable to reproduce this in-house. > However, yesterday, we were finally able to, and I was able to get some > packet dumps at the same time. > > As it turns out, IE seemed to speak a somewhat mangled version of TCP. > Instead of honoring keep-alive timeouts and closing the connection when a > server would send a FIN, it would do the following. > > - Client makes request (success!) > - timeout http-keep-alive is reached, haproxy sends FIN to client > - IE ACKs the FIN, but doesn't send its own FIN (connection is in a > half-open/half-closed state)
That's somewhat common, many clients (often webservice clients) do not monitor their idle connections at all and they don't care about them being closed, dead or in error. It's just a matter of total incompetence of their developers but the end user is directly affected when some requests are sent over a dead connection. > - timeout http-request is reached > - IE tries to send another request on the connection, haproxy replies with > a 408 Here there's something I don't understand : if the FIN was already sent by haproxy, then nothing else can be sent over the connection in that direction. So I understand that you get the 408 in the logs instead, right ? The other possibility is that IE just reads the 408 that was reported *before* the FIN. By the way, I'm thinking about something else : haproxy does not emit 408 when waiting for a second request, precisely to avoid returning an error in this case. It should silently close the connection. Could you indicate what exact version you're using, it's possible that we have a regression or a corner case which is not correctly handled ? > Apparently, as it turns out, IIS plays nice in this situation, and just > returns successful responses. It is not possible after the FIN. From what I remember, IIS does not really apply keep-alive timeout but uses the same request timeout for all requests, but I may be wrong. > Other webservers' possible responses are > either to send a RST, or not respond at all -- in both of those situations, > IE will apparently resend the request. However, in our case, HAProxy is > returning with a 408 (which seems pretty polite to me, and the right thing > to do), and IE isn't bothering to resend the request. > > Two articles that I found that support my theory and findings are: > http://grotto11.com/blog/slash.html?+1039831658 This one is slightly different. > and > http://support.f5.com/kb/en-us/solutions/public/1000/600/sol1672.html This one is closer to what you describe and supports the broken MSIE behaviour. > So, our temporary solution was to keep the http-keep-alive timeout at 10s, > but to raise the http-request time limit to 150s. Not ideal, and it's > definitely causing our concurrent connections to be much higher, but it's > preventing IE users from getting shafted for now. > > So, my question -- is there a current way that I can get around HAProxy > being polite and sending 408 responses? It looks like I could use *option > nolinger*, but that seems a little dirty for what I'm trying to accomplish. Normally it should not do this during keep-alives (except due to a bug). > Aside from that question, I mainly just wanted to report my findings back > to the community. My apologies if this was common knowledge, but it took > long enough for me to track down that I'm thinking it might still be a > mystery to some people. No don't worry, your report is very welcome, because I'm sure you have not made it up, and at the same time it should not happen if we did our job correctly, so let's try to find out exactly what happens to fix it correctly! Thanks, Willy
