Hi, a small follow up to an older thread from November 2015, where massive numbers of epoll_wait calls lead to 100% CPU consumption.
My installation showed the same pattern. As it's also easily reproducible for me with very moderate client traffic (1-10 conns/s), you might maybe be interested in more debug info. For me it usually happens within some minutes into running from a clean start, as soon as a sinlge "rogue" session from a client IP address suddenly pops up with the call counter being at millions within seconds (all other connections / sessions by this client finished correctly). Some points discussed on the other threads related to epoll busy loops: - "option abortonclose" is set, didn't test to disable it, as the old thread mentioned it as not being relevant - "option http-server-close" is set, switched it to "option http-keep-alive" plus "option prefer-last-server" for testing, no change here - "timeout client-fin" and "timeout server-fin" were set, disabling them prevented the quick appearance of new epoll_wait loops As I can very easily reproduce the behaviour with the current config and a very moderate traffic pattern (1-5 conns / s), just let me know, if you'd like to see some other debug info, than what's provided below. Software versions: ---------- $ uname -a Linux xxxlb02 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux haproxy -vv HA-Proxy version 1.6.6 2016/06/26 Copyright 2000-2016 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with OpenSSL version : OpenSSL 1.0.1t 3 May 2016 Running on OpenSSL version : OpenSSL 1.0.1t 3 May 2016 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.35 2014-04-04 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with Lua version : Lua 5.3.1 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. ---------- Used configuration file: ---------- http://paste.ubuntu.com/19362994/ ---------- Running processes: ---------- $ ps afx | grep haproxy 39188 ? Ss 0:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid 39190 ? S 0:00 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds 39191 ? Rs 7:30 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds ---------- Broken session example: --------- 0x7fd3aaa28040: proto=tcpv4 src=80.187.xx.xx:5958 fe=ft-xxx-all be=bk-xxx srv=xxxweb02 ts=04 age=54s calls=64037282 rq[f=804e060h,i=0,an=00h,rx=,wx=,ax=] rp[f=8404c060h,i=0,an=00h,rx=?,wx=29m6s,ax=] s0=[7,0h,fd=2,ex=] s1=[9,190h,fd=35,ex=] exp=? run(nice=0) --------- Strace summary: --------- $ strace -c -p 39191 % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 95.43 0.003260 0 238492 epoll_wait 2.43 0.000083 0 583 94 recvfrom 1.96 0.000067 1 49 49 connect 0.18 0.000006 0 66 getsockopt 0.00 0.000000 0 82 close 0.00 0.000000 0 49 socket 0.00 0.000000 0 616 94 sendto 0.00 0.000000 0 35 shutdown 0.00 0.000000 0 35 getsockname 0.00 0.000000 0 109 setsockopt 0.00 0.000000 0 49 fcntl 0.00 0.000000 0 237 epoll_ctl 0.00 0.000000 0 68 33 accept4 ------ ----------- ----------- --------- --------- ---------------- 100.00 0.003416 240470 270 total --------- Strace sample: --------- $ strace -p 39191 # [...] epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 epoll_wait(0, {}, 200, 0) = 0 # [...] --------- Kind regards Tobias Vau
