Re: nbproc>1, ksoftirqd and response time - just can't get it
Hi! 2011/7/26 Willy Tarreau > > There is an option in Del''s servers to install Intel's 10Gbit > > NIC - it is working a way faster (x3, x5) then broadcom. > > Intel's 10G NICs are fast, but generally hard to tune, you have a hard > trade-off between data rate and latency. Basically you tune them for > large or small packets but not both. Still for DDoS they might be well > suited. > > At least they are well suited for tuning which is only thing one may want when something is not working. > > But anyway it is > > impossible to have one server to filter 10Gbit DDoS attack, at least it > is > > very far from default configuration, so I failed to find solution. > > 10G DDoS is something very hard to resist ! > > But here, in the year 2011 it is not that difficult to run such an attack.:( > > But as haproxy is really good software to handle DDoS you can use > amazon's > > cloud. And that was my final solution. Amazon's servers are taking > 400Mbit > > per node, so installing 60 EC2 nodes I've managed to filter DDoS traffic > at > > 24Gbit/s (It was somewhere around 10*10^6 requests per second, or > ~150-160k > > session rate per node) without any problems. Yes, there was some > balancing > > issues, and some failures for couple of minutes, but it's nothing > comparing > > to typical DDoS expirience. > > Wow! I'm impressed! But did you perform the tests from within Amazon's > cloud > or from the outside ? I don't know what their external peering is, and I'm > wondering how you managed to send 24 Gbps over the internet. Also, what's > the cost of running 60 of these nodes ? > We've tested from the outside. In fact that was a real attack. Botnet consisted of ~10-12k bots each opening 1000 connections/second. There were some failures, when amazon's load balancers responded 'unknown failure', but things were getting back to normal in a few minutes. At peak we had 60 nodes each running at 50Mbyte/s, if you want I can make a screenshots from amazon's console with stats and send it to you. As to the costs... running one hour of such ec2 node is $0.38 * 60 = $22/h, and traffic is $0.13/Gb - this is the most expensive part. > > And question to Willy: What hardware in your opinion is the best to run > > haproxy on to serve lot's of HTTP requests 99% percent of which are trash > to > > be blocked? > > My best results were achieved on Core i5 3.3 GHz (dual core, 4 threads). In > short, I bind network IRQs to core zero and haproxy to core one and the two > remaining threads ensure there is some rope left for system management, > SSH, > etc. With this you can run at 100% CPU all the day if you want. I managed > to > get haproxy to dynamically filter 300k connections per second on such a > configuration. That's SYN, SYN/ACK, ACK, RST, based on a source IP's > connection > rate. > > How may client IPs you were testing? I guess when number of clients reaches some high value there could be problems with CPU cache because tables are growing? > Depending on the type of DDoS you're dealing with, it may be worth > distributing > the NIC's IRQs to multiple cores so that the kernel's work can scale (eg: > emit > a SYN/ACK). It might even be worth having multiple haproxy processes, > because > when you're blocking a DDoS, you don't really mind about monitoring, stats, > health checks, etc... You only want your machine to be as fast as possible, > and > doing so is possible with multi-queues. You then need to spread your NICs > IRQs > to all cores and bind as many haproxy processes as cores (and maually pin > them). > The CPU-NIC affinity will not be good on the backend but that's not the > issue > since you're dealing with a frontend where you want to reject most of your > traffic. > > That's true. I've ended trying to filter attacks on Dell R510 with multiple haproxy and irqbalance running. And about correct statistics on multiple processes - why not use shared memory to store stats? Will it affect performance because transferring memory blocks between cores? > Best regards, > Willy > >
Re : force-persist
Hello Willy,
I have worked about this issue and find out some interesting things about what
was going on. You put me on the right way, thanks for that. In fact, my
application is sending HTTP requests to many web servers (at least 6). Since
the same cookie name was used in HAPROXY for all servers, the cookie value was
set according to server to which HTTP requests was sent. this explain why
affinity between my browser and serv_1 or serv_2 was broken. I have updated
cookie name for each pair of servers (serv_1 & serv_2 have the same fonction,
Serv_3 et Serv_4 also and so on...). Now when setting cookies to check a
specific server, all HTTP requests are sent to the desired server. Thank you
very much for your help.
De : Mouny Jean michel
À : "haproxy@formilux.org"
Envoyé le : Samedi 9 Juillet 2011 0h25
Objet : force-persist
Hello Willy,
I have the following HAProxy configuration installed on a reverse proxy. Behind
this reverse proxy, there are 2 web servers (Serv_1 & Serv_2). If I stopped
Serv_1,
all HTTP requests from my browser are sent to Serv_2 even if cookie
serv1_cookie is set and CHECK_FRONTEND is equal to true in my browser. Since I
used force_persist
option in HAProxy, my HTTP requests should be routed to Serv_1. I don't
understand what"s going on. Please could you help me about this issue?
listen vip_appli01
bind vip_appli01.mycompany.fr:5080
option httplog
option httpclose
option
forwardfor
balance roundrobin
cookie SERVERID insert indirect
acl My_Requets hdr_sub(cookie) CHECK_FRONTEND=true
force-persist if My_Requets
server Serv_1 HAP-Serv_1:80 cookie serv1_cookie check port 80 inter
1 rise 3 fall 3
option httpchk GET /failover/LBfailoverFlag.gif HTTP/1.1\r\nHost:\
192.168.231.13
server Serv_2 HAP-Serv_2:80 cookie serv2_cookie check port 80 inter
1 rise 3 fall 3
+-- Serv_1
Internet Reverse Proxy with Haproxy with force-persist ---|
+-- Serv_2
Re: Increasing amount of spam on the mailing list
On Tue, 26 Jul 2011 08:12:56 +0200, Willy Tarreau wrote: On Mon, Jul 25, 2011 at 10:59:15PM -0700, carlo flores wrote: I love the suggestion and offer to administrate the mail list (and I too volunteer), but, ultimately: whatever. SPAM is part of most any list and the more time the guys spend on one of the best pieces of software in the world, the better. I happily skip these messages in hopes Willy Cyril and the guys never care about wasting their time with mailman and postgix plugins or whatever this list uses. Exactly, when I open the list, I press "D" twice to remove the avg two spams and/or bounce messages and don't even remember what they were. Spam should be ignored, not fought. I personnaly find the haproxy mailing list to be a great training center for my DSPAM installation :) So please don't remove it :D Julien
Re: nbproc>1, ksoftirqd and response time - just can't get it
Hi, > We've tested from the outside. In fact that was a real attack. Botnet > consisted of ~10-12k bots each opening 1000 connections/second. This kind of DDoS seems popular lately. Did it originate from a specific AS, did you try to nullroute? I'm curious because mostly when I see botnet attacks, they are not widely spread throughout the internet but mostly come from <10 AS. Nice to see that AWS performed here, thanks for sharing. :) - Craig
Re: nbproc>1, ksoftirqd and response time - just can't get it
No, this DDoS was from different IPs in 40-50 countries, so it was performed by zombies. As far as I can tell it was performed with Agressor2.0 botnet, successor of DirtyJump3.0. This botnet is constructed to let even 3 y.o. attacker to make serious imact. Each bot not only performs 'http-flood', but also tries to SYN-flood ssh, and all other open ports. And by opening lot's of connections it makes any Cisco ASA completely unusable, as these devices are tracking tcp connections and there are limits, for example Cisco ASA 5510 don't allow for more then 130k connections (so we can say it supports ~100-200 bots). This is why haproxy seems to be the best choice - it allows to control all aspects of communication - swiss army knife. 2011/7/27 Craig > Hi, > > > We've tested from the outside. In fact that was a real attack. Botnet > > consisted of ~10-12k bots each opening 1000 connections/second. > > This kind of DDoS seems popular lately. Did it originate from a specific > AS, did you try to nullroute? I'm curious because mostly when I see > botnet attacks, they are not widely spread throughout the internet but > mostly come from <10 AS. > > Nice to see that AWS performed here, thanks for sharing. :) > > - Craig > >
Re: nbproc>1, ksoftirqd and response time - just can't get it
Hi Dmitriy, On Tue, Jul 26, 2011 at 05:27:10PM +0400, Dmitriy Samsonov wrote: > > Intel's 10G NICs are fast, but generally hard to tune, you have a hard > > trade-off between data rate and latency. Basically you tune them for > > large or small packets but not both. Still for DDoS they might be well > > suited. > > > > > At least they are well suited for tuning which is only thing one may want > when something is not working. Indeed ! > > 10G DDoS is something very hard to resist ! > > > > > But here, in the year 2011 it is not that difficult to run such an attack.:( I agree and that's a real problem. > We've tested from the outside. In fact that was a real attack. Botnet > consisted of ~10-12k bots each opening 1000 connections/second. There were > some failures, when amazon's load balancers responded 'unknown failure', but > things were getting back to normal in a few minutes. At peak we had 60 nodes > each running at 50Mbyte/s, OK so they were opening real connections, not just SYN flooding. SYN flooding generally requires additional system tuning (eg: reduce the synack_retries and adjust the max_syn_backlog to the proper level to send SYN cookies). > if you want I can make a screenshots from > amazon's console with stats and send it to you. Oh yes please do, that would be appreciated. > As to the costs... running one hour of such ec2 node is $0.38 * 60 = $22/h, > and traffic is $0.13/Gb - this is the most expensive part. You mean $0.13 per gigabyte I assume. Indeed this is not cheap. But it can be a good emergency solution when you're experiencing a nasty DDoS that you can't deal with on your side. > > My best results were achieved on Core i5 3.3 GHz (dual core, 4 threads). In > > short, I bind network IRQs to core zero and haproxy to core one and the two > > remaining threads ensure there is some rope left for system management, > > SSH, > > etc. With this you can run at 100% CPU all the day if you want. I managed > > to > > get haproxy to dynamically filter 300k connections per second on such a > > configuration. That's SYN, SYN/ACK, ACK, RST, based on a source IP's > > connection > > rate. > > > How may client IPs you were testing? I guess when number of clients reaches > some high value there could be problems with CPU cache because tables are > growing? It's not important because the ACL is matched only once when the connection is established. Indeed you're right, CPU cache is something very important, but it's already been thrashed with other data structures for each connection. In practice, looking up an IP address within a 1 million entries table takes between 100-300 ns on a recent machine on average. Since 1 million entries do not fit in the cache anymore, I think it's already fast enough for our usage. > > Depending on the type of DDoS you're dealing with, it may be worth > > distributing > > the NIC's IRQs to multiple cores so that the kernel's work can scale (eg: > > emit > > a SYN/ACK). It might even be worth having multiple haproxy processes, > > because > > when you're blocking a DDoS, you don't really mind about monitoring, stats, > > health checks, etc... You only want your machine to be as fast as possible, > > and > > doing so is possible with multi-queues. You then need to spread your NICs > > IRQs > > to all cores and bind as many haproxy processes as cores (and maually pin > > them). > > The CPU-NIC affinity will not be good on the backend but that's not the > > issue > > since you're dealing with a frontend where you want to reject most of your > > traffic. > > > > > That's true. I've ended trying to filter attacks on Dell R510 with multiple > haproxy and irqbalance running. irqbalance only makes things worse for low-latency processes. When you move the IRQs across CPUs only once a second, you can't adapt to such workloads. It's more suited for heavy tasks which are not much sensible to latency. It's much better to manually play with the IRQ masks. > And about correct statistics on multiple > processes - why not use shared memory to store stats? Will it affect > performance because transferring memory blocks between cores? Yes, and more precisely because of locking, because stats are updated at many places. What I want to do is what I did in my old traffic generator, it consists in having one memory area for each process and having the stats dump interface pick and aggregate values from there without locking. If data are correctly accessed, this is not a problem. Multi-word data may still need locking though. The main issue which remains is for rate counters and max thresholds, which still need to be centralized. Another approach I was thinking about was to have a dedicated task in each process responsible for pushing the stats. It would always access valid stats since it's alone when working, and would lock the central ones. Still the issue of rates and maxes persists. Since right now it's already possible to individually ask each process for their stats, it'
Re: Re : force-persist
Hello Jean-Michel, On Tue, Jul 26, 2011 at 06:11:28PM +0100, Mouny Jean michel wrote: > Hello Willy, > > I have worked about this issue and find out some interesting things about > what was going on. You put me on the right way, thanks for that. In fact, my > application is sending HTTP requests to many web servers (at least 6). Since > the same cookie name was used in HAPROXY for all servers, the cookie value > was set according to server to which HTTP requests was sent. this explain why > affinity between my browser and serv_1 or serv_2 was broken. I have updated > cookie name for each pair of servers (serv_1 & serv_2 have the same fonction, > Serv_3 et Serv_4 also and so on...). Now when setting cookies to check a > specific server, all HTTP requests are sent to the desired server. Thank you > very much for your help. Perfect, thank you for your feedback, it's always helpful for other list readers. Cheers, Willy
