Hi, After haproxy starts everything is fine, but after some time one of haproxy processes uses 100% of CPU. I've managed to dump some "show" commands (I have also "show fd", "show sess all" and "strace" but if needed I would prefer to send it privately). If you could give any advice howto debug it further it would be great.
ps aux | grep haprox haproxy 99023 97.5 0.6 3139384 623476 ? Rs 14:07 16:38 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99024 13.0 0.6 3138964 623096 ? Ss 14:07 2:13 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99025 14.1 0.6 3139044 623160 ? Rs 14:07 2:25 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99034 11.9 0.6 3139024 623496 ? Ss 14:07 2:02 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99035 16.3 0.6 3139184 623116 ? Ss 14:07 2:47 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99037 13.0 0.6 3138724 623076 ? Ss 14:07 2:13 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99045 16.3 0.6 3139332 623216 ? Ss 14:07 2:47 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 haproxy 99046 12.0 0.6 3137792 622208 ? Rs 14:07 2:03 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15434 15443 15444 15445 15454 15455 15456 15457 socat /var/run/haproxy/haproxy1.sock - <<< "show info" Name: HAProxy Version: 1.9.6 Release_date: 2019/03/29 Nbthread: 1 Nbproc: 8 Process_num: 1 Pid: 99023 Uptime: 0d 0h12m26s Uptime_sec: 746 Memmax_MB: 0 PoolAlloc_MB: 8 PoolUsed_MB: 8 PoolFailed: 0 Ulimit-n: 1000000 Maxsock: 4012387 Maxconn: 2000000 Hard_maxconn: 2000000 CurrConns: 850 CumConns: 114038 CumReq: 179023 MaxSslConns: 0 CurrSslConns: 4 CumSslConns: 6225 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 161 ConnRateLimit: 0 MaxConnRate: 366 SessRate: 161 SessRateLimit: 0 MaxSessRate: 366 SslRate: 6 SslRateLimit: 0 MaxSslRate: 18 SslFrontendKeyRate: 3 SslFrontendMaxKeyRate: 7 SslFrontendSessionReuse_pct: 50 SslBackendKeyRate: 0 SslBackendMaxKeyRate: 10 SslCacheLookups: 1054 SslCacheMisses: 103 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 ZlibMemUsage: 0 MaxZlibMemUsage: 0 Tasks: 26283 Run_queue: 3 Idle_pct: 39 node: XXXX Stopping: 0 Jobs: 7181 Unstoppable Jobs: 0 Listeners: 6330 ActivePeers: 0 ConnectedPeers: 0 DroppedLogs: 6 BusyPolling: 0 socat /var/run/haproxy/haproxy1.sock - <<< "show activity" thread_id: 0 date_now: 1554380334.734878 loops: 355064374 wake_cache: 2498224 wake_tasks: 352539647 wake_signal: 0 poll_exp: 355037871 poll_drop: 194560 poll_dead: 0 poll_skip: 0 fd_skip: 0 fd_lock: 0 fd_del: 0 conn_dead: 0 stream: 710586336 empty_rq: 15313 long_rq: 57 cpust_ms_tot: 747 cpust_ms_1s: 0 cpust_ms_15s: 0 avg_loop_us: 1 haproxy -vv HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/ Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -DIP_BIND_ADDRESS_NO_PORT=24 OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 Running on OpenSSL version : OpenSSL 1.1.1b 26 Feb 2019 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.5 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE version : 8.42 2018-03-20 Running on PCRE version : 8.42 2018-03-20 PCRE library supports JIT : yes Encrypted password support via crypt(3): yes Built with multi-threading support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTX side=FE|BE h2 : mode=HTTP side=FE <default> : mode=HTX side=FE|BE <default> : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace
