Hi!
I found out that in some situations under high rate of incoming connections
haproxy=2.0.1 starts leaking sockets. It looks like haproxy doesn't close
connections to its backends after request is finished (FIN received from
client) thus leaving its server-sockets in close-wait state.
As an example this simple config starts leaking right from the start for
me. And everything ends with messages: "local0.emerg Proxy
server.local:18400 reached process FD limit (maxsock=4130). Please check
'ulimit-n' and restart."
My config look very similar to this:
global
daemon
uid 120
gid 126
stats socket /var/run/haproxy.sock mode 700 level admin expose-fd
listeners
master-worker
log 127.0.0.1:516 local0 warning
maxconn 2000
tune.ssl.default-dh-param 2048
defaults
log global
maxconn 4096
mode tcp
retries 3
timeout client 1h
timeout connect 5s
timeout server 1h
option redispatch
option dontlognull
listen server.local:18400
bind ipv6@::1:18400 tfo
bind ipv4@127.0.0.1:18400 tfo
mode tcp
balance leastconn
timeout server 24h
timeout client 24h
option dontlog-normal
log 127.0.0.1:516 local1 info
option httpchk GET /check HTTP/1.1\r\nHost:\ server.local
http-check send-state
http-check expect status 200
tcp-request inspect-delay 10s
tcp-request content reject if { nbsrv lt 1 }
default-server weight 50
server backend-server.local:17995 backend-server.local:17995 check port
17994
# haproxy -vvv
HA-Proxy version 2.0.1-1 2019/06/27 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fno-strict-aliasing
-Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare
-Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers
-Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_REGPARM=1 USE_GETADDRINFO=1
USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE
+PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
+REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=32).
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
<default> : mode=HTX side=FE|BE mux=H1
<default> : mode=TCP|HTTP side=FE|BE mux=PASS
Available services :
prometheus-exporter
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
--
Best regards,
Maksim Kupriyanov
