On Mon, Jul 18, 2011 at 06:42:54AM +0400, Dmitriy Samsonov wrote:
> My test setup is three Dell r410 servers (dual Intel(R) Xeon(R) CPU X5650  @
> 2.67GHz - 24 threads total, 128Gb RAM) all connected to 1Gbps network.
> One server is haproxy, configured to block all requests with
> 'Accept-Encoding: none':
> global
>  daemon
> maxconn 80000
> option forwardfor
>  retries 10

Something is missing here above, probably "defaults" and maybe a few other
options before "option forwardfor", such as "mode http" and even a few

> frontend public
>         bind
>         default_backend nginx
>         acl accepts_none hdr(Accept-Encoding) -i none
>         errorfile 403 /raid/emptypage.txt
>         block if accepts_none
> backend nginx
> server srv maxconn 8192
> File /raid/emptypage.txt is an empty file made with 'touch
> /raid/emptypage.txt'.
> I'm doing ab2 -c 1000 -H 'Accept-Encoding: None' -n 1000000
> on two other servers and get following:
> When nbproc = 1 haproxy saturates 100% of cpu core it runs at, but server is
> running nice, I'm able to get reply from nginx behind by using curl on my
> machine: curl, ab reports 16833 requests/second each and
> longest request is around 14seconds.

16k req/s is quite low for such a server, especially since no request should
be forwarded. You should check how the user/system CPU usage is spread, I
suspect there could be an important part spent in the system. Such a test
should better report numbers in the 100k range.

Also, 14 seconds to respond is simply unacceptable. Something is not
working at all. 14 milliseconds would even be high for this type of
processing but could be explained by packet queuing and interrupt
rate depending on the NIC tuning.

> When I change nbproc to higher values (maximum is 24 as there 24 threads
> total) I can see ksoftirq/0 process saturating cpu core, network becomes
> slow on server, ab reports same 16k-17k requests/second for each client, but
> longest request is always around 20-30 seconds.

At such loads, you should not see ksoftirq trigger. 16kreq/s, assuming
there is no keep-alive will basically result in 7 packets per request,
(4 in, 3 out) so you're running around 112 kpps, or 64k IN and 48k out.

Could you check with "vmstat 1" your interrupt rate ? Wouldn't your NIC
be a BNX2 with an old firmware such as 1.9.6 which is known for causing
a high number of packet losses ? (check with ethtool -i eth0).

> I've seen such things with ksoftirq/0 running at 100% and network is almost
> down during DDoS attacks in case of too many iptables rules but what is
> happening now?

Indeed, if you have many iptables rules, then your machine will be very
sensible to SYN floods because for each SYN it will be necessary to find
the matching rule. Do you have ip_conntrack / nf_conntrack loaded ? I
assume that it's the case. Did you tune it ? Default settings are fine
for a desktop usage but not for a server. It is possible that all the
time is wasted in the conntrack hash table lookup.

> And what number to use at nbproc? Is it ok to have haproxy
> running at 100%?

It can be under extreme situations (eg: ddos) but it should not be the
norm, otherwise your scaling abilities are seriously limited. Nbproc
should normally be 1. With proper tuning this will give you much more
raw performance than what you can get in multi-process and poor tuning.
To give you an idea, at Exceliance, we can get 300kconn/s out of a single
core of a dual-core i5 3 GHz. You're at 16k.

What is very important too is to bind network interrupts and processes
to different core sharing the same L2 cache. If you force all your NIC
interrupts to core 0 and bind haproxy to core 1, chances are that they
will be on the same cache (check in /proc/cpuinfo). This is how I get
the most performance, with IRQs taking between 50 and 80% of one core,
and haproxy 100% of another core.

> It looks like I can have 30k requests per second in my
> setup, is there any way to make it higher? I've done some basic tuning like
> tcp_max_tw_buckets = 1024*1024, tcp_tw_reuse = 1, tcp_max_syn_backlog =
> 30000. Am I running out of options?

There are a number of other options, such as net.core.somaxconn which
also limits the backlog (128 by default!). But in your case, the apparent
lack of NIC/process binding can be one cause, and if iptables is there,
it surely is another one.

It is easy to verify if iptables is likely involved : for this you'll
enable keep-alive. Add this to your haproxy config :

     option http-server-close

And replace :

     block if accepts_none

with :

     redirect location / code 302 if accepts_none

Then use "ab -k" instead of "ab" to run your tests. It will try to maintain
connections open as long as possible. The only difference is that it will
see status 302 instead of 403. But that's not a problem for a test.

If you see that the numbers increase by a factor of approximately two, it
will probably mean that you're bound by the packet count and/or by the CPU.
If your numbers increase a lot more (eg: 5-10 times) then it will be very
likely due to the fact that you don't stress conntrack anymore and don't
spend your CPU cycles doing connection lookups.

I really encourage you to try this as it's a simple test. Hint: on my
6 y-o laptop, I'm already getting your numbers above, and testing with
keep-alive doubles the performance.


Reply via email to