Hi all, I'm observing some odd behavior with seamless reloads and processes hanging around. It appears when a reload is triggered with any active client connections, the new process comes up with the proper -sf <pid list>, but the old process(es) is only terminated after the last client disconnects (definitely longer than timeout client) and lives for a very long time in our experience or until a restart of the master process is triggered. I'm using h2 on the front-end with HTX.
Would hard-stop-after apply in this case since the reload signal is USR2, but hard-stop-after is documented to apply to USR1? This is somewhat disconcerting, since it results in an effective memory leak on every reload, which can happen pretty frequently in our setup. If hard-stop-after applies here, then it's not a bug, but perhaps the documentation should clarify its meaning. I first observed this with 1.9.2, it continues in 1.9.3, but the behavior may exist before that. I've included my systemd configuration (from systemctl cat haproxy), in case that's the culprit. Best, Luke — Luke Seelenbinder Stadia Maps | Founder stadiamaps.com # /lib/systemd/system/haproxy.service [Unit] Description=HAProxy Load Balancer Documentation=man:haproxy(1) Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz After=network.target rsyslog.service [Service] EnvironmentFile=-/etc/default/haproxy Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS ExecReload=/bin/kill -USR2 $MAINPID KillMode=mixed Restart=always SuccessExitStatus=143 Type=notify [Install] WantedBy=multi-user.target # /etc/systemd/system/haproxy.service.d/customexec.conf [Service] ExecStartPre= ExecStartPre=-/bin/rm /run/haproxy/global ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS ExecStart= ExecStart=/usr/local/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS ExecReload= ExecReload=/bin/bash -c '/usr/bin/socat /run/haproxy/admin.sock - <<< "show servers state" > /run/haproxy/global' ExecReload=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS ExecReload=/bin/kill -USR2 $MAINPID Output of cat /etc/defaults/haproxy: # Defaults file for HAProxy # # This is sourced by both, the initscript and the systemd unit file, so do not # treat it as a shell script fragment. # Change the config file location if needed #CONFIG="/etc/haproxy/haproxy.cfg" # Add extra flags here, see haproxy(1) for a few options EXTRAOPTS="-x /run/haproxy/admin.sock" haproxy -vv: HA-Proxy version 1.9.3 2019/01/29 - https://haproxy.org/ Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wdate-time -Werror=format-security -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018 Running on OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with Lua version : Lua 5.3.3 Built with network namespace support. Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with PCRE2 version : 10.22 2016-07-29 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with multi-threading support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTX side=FE|BE h2 : mode=HTTP side=FE <default> : mode=HTX side=FE|BE <default> : mode=TCP|HTTP side=FE|BE Available filters : [SPOE] spoe [COMP] compression [CACHE] cache [TRACE] trace
publickey - [email protected] - 0xB23C1E8A.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
