Folks,
I have issues setting up haproxy box.
I'm experimenting with 2 boxes:
10.16.2.98: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz 24 cores, 24gb ram
10.16.2.97: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz16 cores, 12 gb ram
Both boxes run ubuntu 12.04.02 64 bit. I installed haproxy 1.5-dev19 to
10.16.2.98. I also installed nginx from apt repository on both boxes. I
raised number of file handles to 100k via /etc/scurity/limits.conf and
applied following sysctl tuning:
root@elshaproxy002:~# grep -v ^#
/etc/sysctl.d/10-haproxy-network-tuning.conf |grep .
fs.file-max = 331287
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.core.netdev_max_backlog=65536
net.core.somaxconn = 262144
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_mem = 200000 280000 300000
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_max_tw_buckets = 1000000
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_sack = 1
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_synack_retries = 1
net.netfilter.nf_conntrack_max = 10485760
net.nf_conntrack_max = 10485760
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.eth1.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
root@elshaproxy002:~#
Haproxy configuration:
root@elshaproxy001:~# grep -v "^ *#" /etc/haproxy/haproxy.cfg|grep .
global
maxconn 200000
nbproc 24
user haproxy
group haproxy
daemon
defaults
no log
mode http
option dontlognull
retries 3
option redispatch
option tcp-smart-accept
option tcp-smart-connect
maxconn 200000
contimeout 5000
clitimeout 50000
srvtimeout 50000
frontend http
bind :8080
default_backend mercury0
frontend https
default_backend mercury0
mode http
bind 0.0.0.0:8443 ssl crt /etc/haproxy/haproxy.pem
backend mercury0
option http-server-close
server srv5 10.16.2.97:8888 weight 1 maxconn 200000 check inter 4000
root@elshaproxy001:~#
I'm using httpress tool
(https://bitbucket.org/yarosla/httpress/wiki/Home) to hit those boxes
and I run httpress on 9 machines. I run httpress with different levels
of concurrency, each client sends 100 messages, keep-alive is enabled. I
did 4 experiments: hit nginx on box 10.16.2.98, hit nginx on box
10.16.2.98, configured nginx on 10.16.2.98 as
frontend for nginx on 10.16.2.97 and hit 10.16.2.98, configured
haproxy on 10.16.2.98 as frontend for nginx on 10.16.2.97 and hit
10.16.2.98. I have following results:
Requests per second:
concurrency Nginx-at-10.16.2.97 Nginx-at-10.16.2.98 nginx98-nginx97
haproxy98-haproxy97
90 87450 113588 25686 32299
180 58387 133053 39206 40835
360 83948 75648 50321 56404
720 73241 80942 46841 42129
1440 123983 75017 47368 44519
2880 127310 131181 50852 41352
5760 107054 127601 64884 38498
11520 109051 128411 56356 40970
23040 120757 99136 59564 42327
Request time, ms
concurrency Nginx-at-10.16.2.97 Nginx-at-10.16.2.98 nginx98-nginx97
haproxy98-nginx97
90 1.28889 0.788889 3.56667 2.88889
180 4.17778 1.41111 4.63333 4.6
360 4.83333 5.91111 7.24444 6.8
720 14.9333 10.6667 15.7889 17.3889
1440 12.2 27.1778 31.6667 33.5889
2880 23.9333 22.4 64.1444 69.8222
5760 54.7667 48.1556 93.8556 149.778
11520 106.133 98.4667 205.2 281.567
23040 194.811 244.489 391.489 544.711
As you can see nginx outperforms haproxy. Am I missing something
important in configuration? Can haproxy performance be improved?
Thanks,
Kirill.
