Hello Sami, On Wed, Jun 13, 2012 at 10:12:21AM +0200, [email protected] wrote: > Hello, > > We are experiencing since few weeks lots of connections problems on our > haproxy dedicated server. > * > "possible SYN flooding on port 80. Sending cookies."* > > Because of that lots of connections are dropped : > > *"RX packets:851220575 errors:0 dropped:16254629 overruns:0 frame:0 > TX packets:2006236870 errors:0 dropped:0 overruns:0 carrier:0*"
Note that it's saying that packets are dropped, not connections. > We have a huge ammount of legitimate traffic and we have correctly > removed "nf_conntrack" in the kernel. > > Here are also few settings that we have changed in our sysctl.conf > * > net.ipv4.tcp_max_syn_backlog = 819200 > net.ipv4.tcp_synack_retries = 1 > net.ipv4.tcp_syn_retries = 1 > net.core.netdev_max_backlog = 1000* I bet your net.core.somaxconn is set to the default 128. You should significantly increase it (eg around 10-20k). Also, your tcp_max_syn_backlog is set particularly high, you should lower it to around 10-20k too so that during a SYN flood you can quickly switch to SYN cookies. Otherwise you're a SYN amplifier until the backlog is full. > we are using : > * > Centos 5.3 > kernel : 2.6.28.4-xxxx-std-ipv4-32 > haproxy : 1.4.9* > *Intel(R) Xeon(R) CPU W3520 @ 2.67GHz* > > It seems also that connections are dropped when we have more than 50Mb/s > of traffic . > > is there any way to get rid of these Synflood messages and avoid to drop > connections ? SYN cookies are the way to go to resist to high SYN flood rates. It's fine that you lowered the tcp_synack_retries during the event, however you must keep in mind that leaving it this low will inevitably prevent some users from connecting (can't lose more than one SYN/ACK). So you should not leave such a setting enabled all the time. Generally I observed that using 2 here with reasonable backlog (eg: no more than 20k) allows a quick switch to SYN cookies during floods without wasting too much of your bandwidth with the initial SYN/ACK responses. The worst you can do when dealing with a DoS is to make your system behave in a non-standard fashion, because you'll help the attacker ruin your users experience. Better waste a bit of bandwidth than prevent valid visitors from connecting. The SYN flood message is not dramatic, it informs you that the system is switching to SYN cookies for your good. If you have enough outgoing BW, you can disable SYN cookies and keep synack_retries set to 2, but you'll then be a 3x amplifier, ie you'll be sending 150 Mbps of SYN/ACK for 50M SYN received. And you'll need huge backlogs. Note that sending SYN/ACKs consumes CPU, whether it is when looking up the session in the backlog or when building the SYN cookie. So it's important to use a well balanced configuration. Ideally, SYN cookies should only trigger when you know you can't cleanly handle the SYN rate (eg: too much BW usage or too high CPU usage). A few years ago, I was involved in implementing a quick workaround for a SYN flood, we tested the box at 100k and 200k SYN/s. It was a single socket dual-core 1.8G opteron with a gigE NIC. At 200k/s, the machine was only processing SYNs and replying, but the useful traffic was far too low to consider the service available. At 100k/s we still supported a sustained rate of 8k valid connections per second forwarded via haproxy. This was with the settings above (10-20k for somaxconn/max_syn_backlog, and synack_retries to 2). Hoping this helps, Willy
