Wiilly, Lucas, thank you so much for analyzing my configs and your help. We did find out what was wrong.
Some long time ago we added 'option nolinger' to the defaults section. This was figured by trial and error, and that option, on 1.4, served us well to the point of us forgetting about it. When were trying out 1.5, we started experiencing phantom RSTs that, of course, were happening because of nolinger. At which point we said "nolinger is evil and should not be used", and took it out all together. The problems that we were experiencing after version update didn't remind us of the time where we had to put that nolinger in to combat exactly the same symptoms. We rolled the haproxy version back, and of course it didn't make things any better until we put the nolinger back in. I think that in 1.4, nolinger in defaults section only affected backend side, and in 1.5 it started applying to both. I honestly don't quite understand what's wrong with our set up except that: - it's not the application per se, because the application processes the same volume of the requests just fine (when nolinger is turned on) - it's either the network framework (J2EE) settings or kernel settings on the application side that are to fault The failure comes when the session rate reaches some critical volume. At that point we see sharp rise of concurrent sessions (sockets in ESTAB state rise the same), then sharp rise of sockets in SYN-SENT state, and then the whole TCP stack stalls (no impact on existing TCP connections though, like any open SSH shells are not affected to the slightest). There aren't really any problems with unfinished sockets (high number of sockets are only in ESTAB, SYN-SENT and TIME-WAIT states), there is a moderate (few thousand) amount of sockets in client-side TIME_WAIT, but nowhere close to numbers that would cause a problem. So, the stall really occurs on the network side of the application, in turn causing haproxy to stall. I'll see if I can eventually create steps to reproduce that stall in a simple environment, at least to figure out if its haproxy queue that gets stalled, or the whole TCP stack of the machine where haproxy is running, but I don't understand how that scenario would cause the haproxy machine to stop promptly responding to incoming SYNs. I don't think the problem is maxing out maxconn, because our connect timeout is 3s (no explicit queue timeout), and there isn't even a 503 response, and attempting to connect through haproxy during the stall times out way longer than 3s. Thank you, Pawel. On Thu, May 21, 2015 at 5:04 AM, Willy Tarreau <[email protected]> wrote: > Hi Pawel, > > On Tue, May 19, 2015 at 02:47:41PM -0700, Pawel Veselov wrote: > > > This settings should theoretically make > > > haproxy behave exactly the same. > > > > > > > So think that somehow, 1.5 was creating or keeping a lot more open > > connections at a time, and depriving the kernel, or its own limits of > > available connections? > > There's no reason, you should definitely have the same number of > connections with both versions. Also, even on small machines the number > of connections a single process will sustain when reaching servers will > be much lower than the kernel's default FD limits due to the limited > number of source ports. For example, here on a 2GB-equipped atom, the > limit is already at 204k fd, which is much larger than the 2*64k you'd > have from a single source address, and I'm expecting that your production > LB is larger than this box : > > $ cat /proc/sys/fs/file-max > 204282 > > > > In its absence, the behavior would indeed have changed, from the old > > > default: > > > option http-tunnel [1] > > > > > > to: > > > option http-keep-alive [2] > > > > > > > > > One thing I can suggest is to replace "option httpclose" in the default > > > section with the following configuration: > > > option http-keep-alive > > > option prefer-last-server > > > timeout http-keep-alive 5s (or whatever keep alive timeout works for > you) > > > > > > > I personally don't see any reason for having any connections left kept > > alive. We want to minimize the amount of open files, we've had problems > > with keep-alive choking us quite quickly. > > OK that's fair enough. Then you should use http-server-close. "httpclose" > is a hack from version 1.1 which consists in simply adding > "connection: close" to the headers and to let each side decide to close. > In practice it used to result in a number of servers to keep the connection > alive despite this, forcing us to implement extra hacks like "forceclose". > > Since 1.4 this connection management is much cleaner as haproxy can > actively > close a connection, while it was only watching it previously. > > > I see the doc says that httpclose is "...is deprecated since what it does > > is very cheap but not reliable", which I guess means that it won't work > > with all the servers, but our servers honor the Connection:close header > > quite fine, so I don't see a reason to change that. > > Perfect, then just use http-server-close, even on 1.4 since you have it. > > > I guess what I should do - is try 1.5 during "quiet time", and compare > the > > environment (open fds, etc) with 1.4, and see what is different... > > Or maybe first test http-server-close on 1.4 to see if there's anything > related to this. It could very well be that something on the network > doesn't always see the closed connections. > > Regards, > Willy > > -- With best of best regards Pawel S. Veselov
