Hi all, We have been trying to get better performance from our simple haproxy setup since we encountered issues when getting over 15k session/sec.
Our simple test setup to reproduce the issue: * 1 server with 2 CPUs Xeon (E5 core i7) with 6 cores each with hyperthreading on (24 cores total) * 1 Intel I350 Gigabit ethernet interface * CentOS 6.5 kernel 2.6.32-431.11.2.el6.centos.plus.x86_64 * 1 haproxy 1.5.1 recompiled (1.4 from the distribution had the same issues anyway) * 1 nginx on the same host with one simple worker serving a very small static file (< 10 bytes) * 4 client servers running siege with a simple HTTP GET / at concurrency 500 each, in the same VLAN We only get session rate of ~16k per second which seems very low. The interesting symptom is that we get 100% CPU for the haproxy process with very low connection rate. The CPU usage is split this way: 29.3%us, 46.3%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 24.3%si, 0.0%st The very high si and sy seems strange, we focused on soft IRQ config after that to no avail. We tried all the tweaks we could think about: * playing with sysctl (we don't have conntrack, we don't have iptables, tcp settings seems OK) * we tried recompiling the igb driver to get all the options, increasing the rx-usecs setting with ethtool, this reduces the number of hard IRQ (but probably increases latency) but does not impact CPU usage * we looked at the process with strace/perf without much success to get any interesting info The most interesting part: * we did the trick to set the smp_affinity of our eth0 interface to cpu 0 and haproxy on cpu 1 with taskset BUT the soft interrupt CPU stays on cpu 1 (with the haproxy). This is not what is documented from the linux kernel, we dug into the RPS and RFS network features but they are not activated in Centos 6 by default so they should not interfere. I think that we have been looking way too deep in the problem and the solution must be right in front of us. Does anyone have ideas? -- Best, Maxime Brugidou @ Criteo Additional infos: $ haproxy -vv HA-Proxy version 1.5.1 2014/06/24 Copyright 2000-2014 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O2 -march=native -g -fno-strict-aliasing OPTIONS = USE_LINUX_SPLICE=1 USE_LIBCRYPT=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built without OpenSSL support (USE_OPENSSL not set) Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK $ cat /etc/haproxy/haproxy.conf global log /dev/log local1 notice maxconn 8192 user haproxy group haproxy defaults log global mode http retries 0 timeout client 5s timeout connect 1s timeout server 5s option dontlognull option http-server-close option httpchk GET /admin/status option httplog option redispatch option splice-response balance roundrobin frontend main bind 0.0.0.0:80 maxconn 8192 use_backend test backend test server nginx 127.0.0.1:8081 maxconn 8192
