I've got the same problem described by Janusz Dziemidowicz. After enabling h2 on Haproxy 1.8.8 it slowly accumulates frontend connections. Reload or restart cuts them down, but they start to grow again. Upgrade to Haproxy 1.8.9 did not help. See a 40-day graph here - https://pasteboard.co/HnHlKdg.png
Milan Petruzelka ################################################################################################################ ./bin/haproxy -vv HA-Proxy version 1.8.9-83616ec 2018/05/18 Copyright 2000-2018 Willy Tarreau <[email protected]> Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O2 -march=native -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label OPTIONS = USE_LIBCRYPT=1 USE_ZLIB=1 USE_VSYSCALL=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Encrypted password support via crypt(3): yes Built with multi-threading support. Built with PCRE version : 8.38 2015-11-23 Running on PCRE version : 8.38 2015-11-23 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with zlib version : 1.2.8 Running on zlib version : 1.2.8 Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with network namespace support. Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available filters : [SPOE] spoe [COMP] compression [TRACE] trace ################################################################################################################ netstat -aptn|grep haproxy|cut -c 69-80|sort|uniq -c|sort -n 4 LISTEN 20 ESTABLISHED 102 CLOSE_WAIT ################################################################################################################ netstat -apton|grep CLOS|grep hapro|head tcp 7292 0 MY_IP:443 CLIENT_IP:53575 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 211 0 MY_IP:443 CLIENT_IP:50930 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 723 0 MY_IP:443 CLIENT_IP:4873 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 242 0 MY_IP:443 CLIENT_IP:50831 CLOSE_WAIT 5205/haproxy off (0.00/0/0) tcp 1 0 MY_IP:443 CLIENT_IP:18309 CLOSE_WAIT 5205/haproxy off (0.00/0/0) tcp 200 0 MY_IP:443 CLIENT_IP:64678 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 242 0 MY_IP:443 CLIENT_IP:50085 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 242 0 MY_IP:443 CLIENT_IP:64613 CLOSE_WAIT 16694/haproxy off (0.00/0/0) tcp 347 0 MY_IP:443 CLIENT_IP:60401 CLOSE_WAIT 5205/haproxy off (0.00/0/0) tcp 753 0 MY_IP:443 CLIENT_IP:64146 CLOSE_WAIT 16694/haproxy off (0.00/0/0) ################################################################################################################ > show fd 4 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb4bf0 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=GLOBAL 5 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb6df0 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-http 6 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1eb7300 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-http 8 : st=0x05(R:PrA W:pra) ev=0x01(heopI) [nlC] cache=0 owner=0x1ed1b10 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-service 9 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlC] cache=0 owner=0x1ed2020 iocb=0x4be950(listener_accept) tmask=0xffffffffffffffff umask=0xfffffffffffffffe l.st=RDY fe=fe-service 10 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x754758 iocb=0x48cdf0(unknown) tmask=0xffffffffffffffff umask=0xfffffffffffffffe 11 : st=0x05(R:PrA W:pra) ev=0x00(heopi) [nlc] cache=0 owner=0x4e7200 iocb=0x4e7200(thread_sync_io_handler) tmask=0xffffffffffffffff umask=0xfffffffffffffffe 13 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f05c30 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f055a0 14 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ea060 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a1580 15 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1ff94d0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20fa360 16 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b7a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fc58f0 17 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2012970 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a7480 18 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b7e30 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x210b610 19 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1efa590 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f02610 20 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b83a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x22a2800 21 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0f10 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x216ecf0 22 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f02a40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20d03b0 23 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8e80 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22d2ff0 24 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8f2c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1ee8030 25 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f88c90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fa1e20 26 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8cb0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f2d260 27 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20d0b40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f04870 28 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a3670 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f02790 29 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21639c0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f21dd0 30 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x217cac0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22cd210 31 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2163540 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x209b2e0 32 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x209b750 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20a02b0 33 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20ce280 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2321d60 34 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2162d40 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1fbe580 36 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f05a70 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20c0d80 37 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20e43a0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1f18200 38 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20a0120 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x20ea4e0 39 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2082a90 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a1e00 40 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x21b8740 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x22a5ca0 41 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x215a730 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x236f300 42 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x210b440 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x2326220 43 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20d8110 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x1faa770 44 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x216f020 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x20e7850 45 : st=0x24(R:Pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x20816b0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x1 cflg=0x00201300 fe=GLOBAL mux=PASS mux_ctx=0x1fa6600 46 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x2011390 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x1f2bf70 49 : st=0x20(R:pra W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8de50 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203300 fe=fe-http mux=H2 mux_ctx=0x2327030 50 : st=0x25(R:PrA W:pRa) ev=0x00(heopi) [nlc] cache=0 owner=0x1f8aea0 iocb=0x4d2c80(conn_fd_handler) tmask=0x1 umask=0x0 cflg=0x80203306 fe=fe-http mux=H2 mux_ctx=0x230db80 ################################################################################################################ haproxy.cfg global daemon user haproxy group haproxy stats socket /some_path/cmd.sock mode 600 level admin stats timeout 1m maxconn 100000 log 127.0.0.1:1234 local0 defaults log global option httplog mode http option http-keep-alive option forwardfor timeout connect 5s retries 3 option redispatch timeout queue 30s ## how long to wait in backend / server queue timeout client 31s ## inactivity timeout during request reading timeout server 120s ## inactivity timeout during response reading timeout tunnel 1h ## long lived connections (WebSocket / TCP) timeout http-request 5s ## max. request duration - headers without post data timeout http-keep-alive 500ms ## timeout when waiting for next request in keepalive mode frontend fe-http maxconn 10000 bind *:80 bind *:443 ssl no-sslv3 alpn h2,http/1.1 crt /some_path/ai1.pem crt /some_path/default.pem capture cookie X-Sid len 45 capture request header Host len 30 capture request header X-Forwarded-For len 50 capture response header X-Rid len 30 capture response header X-Cache len 20 default_backend be-nginx01 backend be-nginx01 default-server inter 5s rise 2 fall 3 option httpchk HEAD / server nginx01 unix@/some_path/listen-auto.sock check maxconn 30
