Hi, On Mon, Jul 18, 2011 at 06:42:54AM +0400, Dmitriy Samsonov wrote: > My test setup is three Dell r410 servers (dual Intel(R) Xeon(R) CPU X5650 @ > 2.67GHz - 24 threads total, 128Gb RAM) all connected to 1Gbps network. > > One server is haproxy, configured to block all requests with > 'Accept-Encoding: none': > > global > daemon > maxconn 80000 > option forwardfor > retries 10
Something is missing here above, probably "defaults" and maybe a few other options before "option forwardfor", such as "mode http" and even a few timeouts. > frontend public > bind 192.168.0.1:80 > default_backend nginx > acl accepts_none hdr(Accept-Encoding) -i none > errorfile 403 /raid/emptypage.txt > block if accepts_none > > backend nginx > server srv 127.0.0.1:80 maxconn 8192 > > File /raid/emptypage.txt is an empty file made with 'touch > /raid/emptypage.txt'. > > I'm doing ab2 -c 1000 -H 'Accept-Encoding: None' -n 1000000 > http://192.168.0.1/ on two other servers and get following: > > When nbproc = 1 haproxy saturates 100% of cpu core it runs at, but server is > running nice, I'm able to get reply from nginx behind by using curl on my > machine: curl http://192.168.0.1/, ab reports 16833 requests/second each and > longest request is around 14seconds. 16k req/s is quite low for such a server, especially since no request should be forwarded. You should check how the user/system CPU usage is spread, I suspect there could be an important part spent in the system. Such a test should better report numbers in the 100k range. Also, 14 seconds to respond is simply unacceptable. Something is not working at all. 14 milliseconds would even be high for this type of processing but could be explained by packet queuing and interrupt rate depending on the NIC tuning. > When I change nbproc to higher values (maximum is 24 as there 24 threads > total) I can see ksoftirq/0 process saturating cpu core, network becomes > slow on server, ab reports same 16k-17k requests/second for each client, but > longest request is always around 20-30 seconds. At such loads, you should not see ksoftirq trigger. 16kreq/s, assuming there is no keep-alive will basically result in 7 packets per request, (4 in, 3 out) so you're running around 112 kpps, or 64k IN and 48k out. Could you check with "vmstat 1" your interrupt rate ? Wouldn't your NIC be a BNX2 with an old firmware such as 1.9.6 which is known for causing a high number of packet losses ? (check with ethtool -i eth0). > I've seen such things with ksoftirq/0 running at 100% and network is almost > down during DDoS attacks in case of too many iptables rules but what is > happening now? Indeed, if you have many iptables rules, then your machine will be very sensible to SYN floods because for each SYN it will be necessary to find the matching rule. Do you have ip_conntrack / nf_conntrack loaded ? I assume that it's the case. Did you tune it ? Default settings are fine for a desktop usage but not for a server. It is possible that all the time is wasted in the conntrack hash table lookup. > And what number to use at nbproc? Is it ok to have haproxy > running at 100%? It can be under extreme situations (eg: ddos) but it should not be the norm, otherwise your scaling abilities are seriously limited. Nbproc should normally be 1. With proper tuning this will give you much more raw performance than what you can get in multi-process and poor tuning. To give you an idea, at Exceliance, we can get 300kconn/s out of a single core of a dual-core i5 3 GHz. You're at 16k. What is very important too is to bind network interrupts and processes to different core sharing the same L2 cache. If you force all your NIC interrupts to core 0 and bind haproxy to core 1, chances are that they will be on the same cache (check in /proc/cpuinfo). This is how I get the most performance, with IRQs taking between 50 and 80% of one core, and haproxy 100% of another core. > It looks like I can have 30k requests per second in my > setup, is there any way to make it higher? I've done some basic tuning like > tcp_max_tw_buckets = 1024*1024, tcp_tw_reuse = 1, tcp_max_syn_backlog = > 30000. Am I running out of options? There are a number of other options, such as net.core.somaxconn which also limits the backlog (128 by default!). But in your case, the apparent lack of NIC/process binding can be one cause, and if iptables is there, it surely is another one. It is easy to verify if iptables is likely involved : for this you'll enable keep-alive. Add this to your haproxy config : option http-server-close And replace : block if accepts_none with : redirect location / code 302 if accepts_none Then use "ab -k" instead of "ab" to run your tests. It will try to maintain connections open as long as possible. The only difference is that it will see status 302 instead of 403. But that's not a problem for a test. If you see that the numbers increase by a factor of approximately two, it will probably mean that you're bound by the packet count and/or by the CPU. If your numbers increase a lot more (eg: 5-10 times) then it will be very likely due to the fact that you don't stress conntrack anymore and don't spend your CPU cycles doing connection lookups. I really encourage you to try this as it's a simple test. Hint: on my 6 y-o laptop, I'm already getting your numbers above, and testing with keep-alive doubles the performance. Regards, Willy
