Hi,
I saw in the 2.6.10 release notes to report any issues that seem like
they could be related to the concurrency changes. When reloading config
on 2.6.10 or 2.6.11 on FreeBSD 13.1-RELEASE the old process does not
exit and starts to use 100%+ CPU. This does not happen on 2.6.9 with
the same config.
As a test I set nbthread=1 on 2.6.11 instead of the 40 default on this
system and cannot reproduce the problem. Reloads are also good on test
VMs with only 1 core. Trying different nbthread values in increments of
2 up to 40 gave mixed results, sometimes reloads are OK and other times
the old process has the issue with the same nbthread value that was OK on
a previous test. Higher nbthread values seem to more reliably fail and
my original value of 36 always fails with active traffic.
Everything else runs perfectly on 2.6.11 except this issue during reloads
and the workaround is to just manually kill -9 the old process.
Here is an example after a few config reloads while making changes (with
nbthread=36):
-------------------------------
$ ps auxwww | grep haproxy
root 3642 1437.6 0.5 7736200 169152 - Rs 21:51 28:02.48
/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p
/var/run/haproxy.pid -sf 44929
root 64199 1337.5 0.4 7693192 144848 - Rs 21:29 1200:18.20
/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p
/var/run/haproxy.pid
root 44929 1327.8 0.4 7693192 146672 - Rs 21:40 816:53.98
/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p
/var/run/haproxy.pid -sf 64199
root 88787 0.0 0.3 7674760 114224 - Ss 22:31 0:02.77
/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p
/var/run/haproxy.pid -sf 3642
$
last pid: 10527; load averages: 99.66, 81.61, 73.34 up 0+01:21:05 22:32:20
108 processes: 6 running, 102 sleeping
CPU: 91.8% user, 0.0% nice, 8.2% system, 0.0% interrupt, 0.0% idle
Mem: 528M Active, 211M Inact, 1093M Wired, 345M Buf, 29G Free
ARC: 9216B Total, 9216B Header
Swap: 4096M Total, 4096M Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
44929 root 36 82 0 7513M 143M RUN 6 809:37 1390.71% haproxy
3642 root 36 82 0 7555M 165M RUN 5 20:13 1304.75% haproxy
64199 root 36 82 0 7513M 141M RUN 1 19.9H 1292.72% haproxy
52227 root 9 52 0 46M 4932K uwait 3 0:56 5.66% nscd
88787 root 36 20 0 7495M 109M kqread 1 0:02 2.01% haproxy
-------------------------------
Some ktrace output from one of the old processes is below, I can send
the full ~130MB dump to a developer if helpful.
Thanks!
44929 haproxy CALL clock_gettime(0xe,0x7fffdcde4ea0)
44929 haproxy CALL clock_gettime(0xe,0x7fffdbbdbea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbddcea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffde1eeea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x40,0,0,0x877902a40,0xc8,0x7fffde1eef00)
44929 haproxy CALL kevent(0x5b,0,0,0x8da501040,0xc8,0x7fffdbddcf00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy CALL kevent(0x43,0,0,0x885700900,0xc8,0x7fffdddecf00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdddeceb8)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy CALL kevent(0x84,0,0,0x960d04fc0,0xc8,0x7fffdbbdbf00)
44929 haproxy CALL kevent(0x4f,0,0,0x8af5014c0,0xc8,0x7fffdcde4f00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbbdbeb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbbdbea0)
44929 haproxy CALL clock_gettime(0xe,0x7fffde1eeeb8)
44929 haproxy RET clock_gettime 0
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffde3efea0)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x34,0,0,0x84e500e00,0xc8,0x7fffdeff5f00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy STRU struct kevent[] = { }
44929 haproxy CALL clock_gettime(0xe,0x7fffdcde4eb8)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5eb8)
44929 haproxy CALL clock_gettime(0xe,0x7fffdcde4ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x34,0,0,0x84e500e00,0xc8,0x7fffdeff5f00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5eb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x34,0,0,0x84e500e00,0xc8,0x7fffdeff5f00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5eb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdeff5ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x84,0,0,0x960d04fc0,0xc8,0x7fffdbbdbf00)
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x81,0,0,0x958d03100,0xc8,0x7fffdbfddf00)
44929 haproxy CALL clock_gettime(0xe,0x7fffdf7f9ea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x2e,0,0,0x838b00040,0xc8,0x7fffdf7f9f00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddeb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x81,0,0,0x958d03100,0xc8,0x7fffdbfddf00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddeb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x81,0,0,0x958d03100,0xc8,0x7fffdbfddf00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
44929 haproxy RET kevent 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddeb8)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL clock_gettime(0xe,0x7fffdbfddea0)
44929 haproxy RET clock_gettime 0
44929 haproxy CALL kevent(0x81,0,0,0x958d03100,0xc8,0x7fffdbfddf00)
44929 haproxy STRU struct kevent[] = { }
44929 haproxy STRU struct kevent[] = { }
$ haproxy -vv
HAProxy version 2.6.11-3c6be69 2023/03/17 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.11.html
Running on: FreeBSD 13.1-RELEASE-p6 FreeBSD 13.1-RELEASE-p6 GENERIC amd64
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall
-Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits
-Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option
-Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
-DFREEBSD_PORTS
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1
USE_LUA=1 USE_ACCEPT4=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 -BACKTRACE +CLOSEFROM +CPU_AFFINITY -CRYPT_H
-DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE +LIBCRYPT
-LINUX_SPLICE -LINUX_TPROXY +LUA -MEMORY_PROFILING -NETFILTER -NS
-OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL
+PROCCTL +PROMEX -QUIC -RT -SLZ -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO
+THREAD -THREAD_DUMP +TPROXY -WURFL +ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=40).
Built with OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022
Running on OpenSSL version : OpenSSL 1.1.1o-freebsd 3 May 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.12
Running on zlib version : 1.2.12
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with PCRE2 version : 10.40 2022-04-14
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 13.0.0 ([email protected]:llvm/llvm-project.git
llvmorg-13.0.0-0-gd7b669b3a303)
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
$
