Are you running irqbalance? It may help distribute network interface interrupts
Sent from my iPhone On 18 Jul 2011, at 03:42, Dmitriy Samsonov <dmitriy.samso...@gmail.com> wrote: > My test setup is three Dell r410 servers (dual Intel(R) Xeon(R) CPU X5650 @ > 2.67GHz - 24 threads total, 128Gb RAM) all connected to 1Gbps network. > > One server is haproxy, configured to block all requests with > 'Accept-Encoding: none': > > global > daemon > maxconn 80000 > option forwardfor > retries 10 > > frontend public > bind 192.168.0.1:80 > default_backend nginx > acl accepts_none hdr(Accept-Encoding) -i none > errorfile 403 /raid/emptypage.txt > block if accepts_none > > backend nginx > server srv 127.0.0.1:80 maxconn 8192 > > File /raid/emptypage.txt is an empty file made with 'touch > /raid/emptypage.txt'. > > I'm doing ab2 -c 1000 -H 'Accept-Encoding: None' -n 1000000 > http://192.168.0.1/ on two other servers and get following: > > When nbproc = 1 haproxy saturates 100% of cpu core it runs at, but server is > running nice, I'm able to get reply from nginx behind by using curl on my > machine: curl http://192.168.0.1/, ab reports 16833 requests/second each and > longest request is around 14seconds. > > When I change nbproc to higher values (maximum is 24 as there 24 threads > total) I can see ksoftirq/0 process saturating cpu core, network becomes slow > on server, ab reports same 16k-17k requests/second for each client, but > longest request is always around 20-30 seconds. > > I've seen such things with ksoftirq/0 running at 100% and network is almost > down during DDoS attacks in case of too many iptables rules but what is > happening now? And what number to use at nbproc? Is it ok to have haproxy > running at 100%? It looks like I can have 30k requests per second in my > setup, is there any way to make it higher? I've done some basic tuning like > tcp_max_tw_buckets = 1024*1024, tcp_tw_reuse = 1, tcp_max_syn_backlog = > 30000. Am I running out of options?