Hi Willy, Thanks for having a look at this! > On Thu, Jan 12, 2017 at 11:34:44AM +0000, Carl Pettersson (BN) wrote: > > Hi, > > We have a haproxy setup consisting of a pair of nodes with keepalived, > > which then utilize the proxy protocol to pass requests (roundrobin) to > > a second pair of haproxy nodes. The first pair mainly terminates SSL > > and serves as a highly available entrypoint, while second pair does > > all the logic around routing to the correct application. > > Yesterday at 21:30:35 CET the active node of the first pair suddenly > > accumulated thousands of orphaned sockets. Historically there have > > been around 300 orphans at any time on this machine. A few seconds > > later, the cpu shot up to 100%, up from 30%. At this point, most > > requests started timing out. At the moments just before, we were > > handling 400-450 req/s. The second pair saw no increase of load during > > these problems. > > As far as we can tell so far (investigations are ongoing), there was > > no change anywhere in the environment for several hours preceding this > > sudden activity. We are lucky enough that a large part of the traffic > > coming in is diagnostic data, which we can live without for a while, > > and when we shut down that application, the situation returned to > > normal. Before doing that, we tried both failing over to the other > > node, as well as both restarting haproxy and a full reboot of the > > node. Neither of these helped, the situation returned in less than a minute. > > > > We had a lot of "kernel: [ 1342.944691] TCP: too many orphaned sockets" in > > our logs, as well as a few of these: > > kernel: [2352025.865855] TCP: request_sock_TCP: Possible SYN flooding on > > port 443. Sending cookies. Check SNMP counters. > > kernel: [2352068.014861] TCP: request_sock_TCP: Possible SYN flooding on > > port 80. Sending cookies. Check SNMP counters. > > > > According to the Arbor our hosting provider has, there was no SYN > > attack, however. The servers are fronted by Akamai, which by default > > does two retries on connection failures or timeouts, so this may have > > amplified our problems. > > This feature will be turned off. > > > > So, some questions: > > 1. Does it seem reasonable that the orphaned socket could cause this > > behaviour, or are they just a symptom? > > No, orphans have no such effect and your number was so low that it cannot be > related. Usually you see them in tends to hundreds of thousands. In your case > the two events were clearly unrelated when you compare the log dates (27 days > apart).
Ah, this is due to poor information on my part. The "Too many orphaned sockets" log was taken after the reboot I mentioned, while the "SYN flood" message was just before the reboot. So in reality they are just ~5 minutes apart (the dangers of relative timestamps). Here is a graph showing sockets in use/orphaned and cpu usage: https://cdn.pbrd.co/images/lfCUluWzj.png. We can see that the number of sockets in use sharply increases at 21:30, and the cpu load follows just a short while afterwards. The orphaned sockets fluctuate a bit, but from what you say, those are probably not relevant. > > > > 2. What causes the orphaned sockets? Could haproxy start misbehaving > > when it is starved for resources? > > No, simply a connection which was closed and for which the client has not yet > ACKed the final data. It's a perfectly normal case. However if you see the > message often, it means your number of orphans is too low, and you can > increase it in /proc/sys/net/ipv4/tcp_max_orphans. > Are there any guidelines on scaling this number? As I understand it, the reason that this is limited in the first place is to avoid some simple attacks, but those may be set for different cases than a public facing load balancer? > > 3. We were speculating that it could somehow be related to keepalives > > not being terminated properly, any merit to that thought? > > No, I think based on your log mentionning a SYN flood that in fact you might > have got a "massive" number of SSL handshakes causing the CPU saturation for > a while and the impossibility to process incoming connections for a short > time. Once the somaxconn limit is reached, the system is forced to drop new > SYNs and will log that it was under something looking like a SYN flood. And > depending on the sizing of your machine, your permitted ciphers, your cert > size etc, the cost of a handshake can be high enough for a moderate > connection rate to do this while passing under the radar. > > > The nodes are virtual machines running Centos 7 and haproxy 1.6.7, > > single core with 2 GB memory. As I mentioned, they have been handling > > this load without any hiccups for several months, but we are still > > considering increasing the specs. Would a few more cores have helped, > > or would it just have taken a few more seconds to chew up? > > The danger wiht virtual machines is that their CPU can be stolen by other > machines or by the hypervisor without them noticing, resulting in any pending > traffic to be suddenly delivered as a burst and making this problem possibly > worse. Are you sure you don't have memory ballooning, sub-CPU allocation nor > any such horrors ? Indeed. We did have a look at those numbers, and there was no cpu steals reported at this time (we have indeed seen the problem on other machines at other times, however). Memory ballooning is not used on the installation, thankfully. > > If the issue was only SSL handshakes, extra cores might have helped, till the > point the attacker was able to reach your capacity. If it was something else, > there's little point adding extra cores if the machine is never loaded. > > By the way, how did the problem stop, and how long after it started ? We had the luxury of being able to stop the problem by disabling the incoming calls. The majority of the inbound traffic on this system at the moment is diagnostic data, which we can live without for a few days if it affects the stability of the platform as a whole (We did want to re-enable it today to see if the problems would resume, but due to business reasons the risk of destabilizing could not be taken until later this night). > > > Below is the haproxy configuration. > > (...) > > I'm not seeing anything wrong in it. One thing however, the default maxconn > for a frontend is 2000. If you have 400 conn/s, it means that you will > saturate this once this connections last 5 seconds or more. It's possible you > had a hiccup on the server side slowing down connections an making you reach > the limit on the front. You should increase this (in the defaults section for > example) as well as the global.maxconn value. Yes, maxconn is one of the settings we realized was much too low as part of the investigations. We've seen quite a spectrum of advice on the internet on what values to use - from "somewhere around 4096" to "just put it to 55 million"... We will also be scaling up to 2 or 4 cores which I guess means we'll have to also look into setting nbproc and possibly cpu mappings? > > Regards, > Willy >
