Hi, we've been using haproxy 1.5 for quite a while, and haven't really
run into any major issues until we upgraded from dev24 to dev25.
Starting with dev25 we saw an issue where haproxy doesn't seem to be
reliably closing connections after sending a response if the client
uses keepalive. The same happens with dev26.
Unfortunately I'm unable to replicate the issue on our test servers,
but what we see happen on production is that the number of open
connections slowly rises over time and the haproxy processes use more
and more memory until the OOM killer starts killing them. If we reload
haproxy, the old process will then stick around until it's manually
killed.
I did a tcpdump on the server for 3 minutes or so (much longer than
the 50 second client timeout) and found at least one connection that
made a request, then hung around without any traffic at all for the
rest of the tcpdump. I'm sure there are many more, but it was very
slow looking through the file with wireshark due to its size.
A sanitized and simplified version of our config is below. If there's
any other info that would help I'd be more than happy to try to gather
it. If it involves gdb or anything I'll probably need pretty specific
instructions though. :)
Thanks much
global
log /dev/log local1 info
stats socket /var/run/haproxy.sock
stats bind-process 1
maxconn 100000
user haproxy
group haproxy
nbproc 1
daemon
defaults
mode http
option dontlognull
option allbackups
retries 3
option redispatch
option log-health-checks
maxconn 100000
timeout connect 5s
timeout queue 50s
timeout client 50s
timeout server 50s
timeout tarpit 20s
listen stats1 0.0.0.0:1999
bind-process 1
mode http
maxconn 10
stats enable
stats hide-version
stats show-legends
stats refresh 5s
stats uri /
frontend webservices
log global
bind *:80
bind *:443 ssl crt /etc/haproxy/wildcard-cert.pem ciphers
RC4-SHA:HIGH:!ADH:!SSLv2
bind :::80 v6only
bind :::443 ssl crt /etc/haproxy/wildcard-cert.pem ciphers
RC4-SHA:HIGH:!ADH:!SSLv2 v6only
option forwardfor header X-Forwarded-For-Internal
option http-server-close
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\
%B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\
%[dst_port]
capture request header Host len 40
block unless METH_GET or METH_POST or METH_HEAD or METH_OPTIONS
default_backend static-servers
backend static-servers
balance leastconn
option httpchk GET /haproxy_check/static
http-check disable-on-404
#Remove version prefix
reqrep ^([^\ ]*)\ /v/\w+/(.*) \1\ /\2
server be1-static-prv 10.0.0.1:8080 maxconn 500 check observe layer7
server be2-static-prv 10.0.0.2:8080 maxconn 500 check observe layer7
