Hi All,

I'm looking in a strange issue I'm having and I start to think it is HAProxy related.

I have a setup with HAProxy serving multiple frontends and multiple backends which are Nginx server with PHP-FPM. Sometimes all of the sudden the maxconn limit is hit and connections get queued to a backend server and I do not have a clue why. The backend is not overloaded, not traffic is flowing, Nginx/PHP-FPM picks op other connections like the health checks from HAProxy or out monitoring server, PHP-FPM is not doing anything so no long runing processes, Nginx is doint nothing, but it does not receive any new connection from HAProxy. Sometimes this is for 1 second, but this also happens for as much as 30 seconds.

It does not happen on all backend servers at once, just random at one server. So if I have defined a backend with 2 servers it happens to only one at a time.

I'm running HAProxy 2.0.13 on Debian Buster in a VM. I've tested with 'no option http-use-htx' and HAProxy 2.1.3 and I see the problem on both. Backends are Nginx with PHP-FPM and only using HTTP/1.1 over port 80, also VM's.

Today I disabled H2 on the frontends and now the problem seems to have disappeared. So it seems to be releated to that part. But, I'm not sure. How should I go on and debug this?

The config looks a bit like this (very redacted and very, very much shortened):

global
        master-worker
        log     /dev/log        local0
        log     /dev/log        local1 notice

        daemon
        user            haproxy
        group           haproxy
        maxconn         32768
        spread-checks   3
        nbproc          1
        nbthread        4
        stats socket    /var/run/haproxy.stat mode 666 level admin

# ciphers generator (https://mozilla.github.io/server-side-tls/ssl-config-generator/)
        ssl-server-verify       none

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.2
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

        ssl-dh-param-file /etc/haproxy/ffdhe3072.pem

defaults
        log                     global
        timeout check           2s
        timeout client          60s
        timeout connect         10s
        timeout http-keep-alive 4s
        timeout http-request    15s
        timeout queue           30s
        timeout server          60s
        timeout tarpit          120s

        errorfile 400 /etc/haproxy/errors.loc/400.http
        errorfile 403 /etc/haproxy/errors.loc/403.http
        errorfile 500 /etc/haproxy/errors.loc/500.http
        errorfile 502 /etc/haproxy/errors.loc/502.http
        errorfile 503 /etc/haproxy/errors.loc/503.http
        errorfile 504 /etc/haproxy/errors.loc/504.http

        option http-use-htx

listen admin
        bind            x.x.x.x:8080 ssl crt /some/path/ strict-sni alpn 
h2,http/1.1
        mode            http
        stats enable
        stats uri       /haproxy?stats
        stats auth      username:password
        stats admin if TRUE
        stats refresh 5s

frontend cluster1-in
        # LB itself
        bind x.x.x.x:80 transparent
        bind aaa:aaa:aaa::a:80 transparent
        bind x.x.x.x:443 transparent ssl crt /some/path/
        bind aaa:aaa:aaa::a:443 transparent ssl crt /some/path/

        # Mass hosting VIP
        bind y.y.y.y:80 transparent
        bind aab:aab:aab::a:80 transparent

bind y.y.y.y:443 transparent ssl crt /some/cert.pem crt /another/cert.pem crt /some/path/ strict-sni alpn h2,http/1.1 bind aab:aab:aab::a:443 transparent ssl crt /some/cert.pem crt /another/cert.pem crt /some/path/ strict-sni alpn h2,http/1.1

        mode http
        maxconn 8192

        option httplog
        option dontlog-normal
        option http-ignore-probes
        option forwardfor

        capture request header Host             len 64
        capture request header User-Agent       len 16
        capture request header Content-Length   len 10
        capture request header Referer          len 256
        capture response header Content-Length  len 10

        #
        # Some security stuff starts here
        #
        acl name src -f /some/file.txt

        http-request deny if name
        http-request del-header Proxy
        http-request set-header X-Forwarded-Proto https if { ssl_fc }
        http-request set-header X-Forwarded-Ssl on if { ssl_fc }

        http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
        http-request set-header X-Forwarded-Ssl off if !{ ssl_fc }

http-response add-header Strict-Transport-Security "max-age=31536000;" if { ssl_fc }

        use_backend     
%[req.hdr(host),lower,regsub(^www\.,,i),map(/path/to/map/filename.map,default-cluster)]
        default_backend default-cluster

backend some-backend
        fullconn        4096
        mode            http

        balance roundrobin

        option  abortonclose
        option  prefer-last-server
        option  redispatch
        option  httpchk GET /php-fpm-ping HTTP/1.0
        http-check expect status 200

default-server weight 100 agent-check agent-port 8081 agent-inter 20s check inter 2s rise 3 fall 3 slowstart 5m maxconn 50
        server name1 abc:abc:abc::1:80 cookie name1
        server name2 abc:abc:abc::2:80 cookie name2

        # Sorry Server
        server outage 127.0.0.1:80 backup

        retries 1


Regards,

Sander Klein





Reply via email to