I am currently trying to migrate a somewhat over-complicated and
over-provisioned setup to something simpler and more efficient. The
application servers lots of small HTTP requests (often 204 or 3xx), and
gets requests from all sorts of oddly behaving clients or
over-aggressive pre-connecting user agents. (Thanks for the help in the
client timeout thread!)
The servers are:
* AMD Opteron 4184, 6 cores, 2.8 Ghz
* 2x gigabit Intel 82576 with the igb driver
* centos 6 (2.6.32-358.23.2.el6.x86_64)
Running 1.4.24 with all traffic currently coming from the outside on
eth0 and going to the backend servers on eth1.
The setup that I thought would work well is:
* core 0: all network interrupts
* core 1: haproxy
* core 2-5: httpd for ssl termination
After some adventures in interrupt tunning so that core 0 would not be
killed by > 100k/s we got this to satisfy roughly 20k req/s. At this
point hatop would report the Queue was growing without bond and maxconn
would fill up. The cause looks simple enough, core 1 was consistently
0.00% idle at that load http://pastebin.com/kuT5uCtP.
My first question is if the roughly 1:3 usr/sys ratio is "normal" for a
well functioning haproxy serving tiny http objects? See sample config
attached. While I don't have an empirical basis for comparison on this
older hardware, 20k req/s also "seemed" low.
Because it's what we had used in the old setup I also tried setting
nbproc=6 with no IRQ pinning (tests showed IRQs on one core was
beneficial with no nbproc). This was able to handle close to *twice* as
many req/s. This was a surprising result because almost every thread
mentioning nbproc suggests not using it and I expected at best marginal
gains. Either way I'd prefer to avoid nbproc for stats and all of the
typical reason.
Finally assuming the single process performance can not be further
improved I was considering the following setup:
* core 0: eht0 interrupts
* core 1: haproxy bound to eth0
* core 2: eth1 interrupts
* core 3: haproxy bound to eth1
* core 4-5: ssl terminator
But I could not find too many examples of similar setups and was unsure
if it was a viable long term configuration.
global
log 127.0.0.1 local4 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
daemon
stats socket /var/run/haproxy.socket mode 766
maxconn 65530
defaults
mode http
log global
option dontlog-normal
option dontlognull # uncomment for details
option httplog
option httpclose
option contstats
timeout client 7s
timeout server 4s
timeout connect 4s
timeout http-request 7s
maxconn 65530
listen stats
bind *:81
stats enable
stats uri /ha-stats
frontend foo_in
bind *:80
mode http
default_backend foo_nodes
log global
option forwardfor except 127.0.0.1
backend foo_nodes
balance roundrobin
option tcp-smart-connect
option httpchk HEAD /live-lb HTTP/1.0
server s0 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
server s1 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
server s2 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
server s3 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
server s4 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
server s5 xx.xx.xx.xx:8080 maxconn 4096 maxqueue 1024 check inter 1s fall 5
rise 2
haproxy -vv
HA-Proxy version 1.4.24 2013/06/17
Copyright 2000-2013 Willy Tarreau <[email protected]>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing
OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Available polling systems :
sepoll : pref=400, test result OK
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 4 (4 usable), will use sepoll.
