I might have found a potentially critical bug in haproxy. It occurs when
haproxy is retrying to dispatch a request to a server. If haproxy fails
to dispatch a request to a server that is either up or has no health
checks enabled it dispatches the request to a random server on any
backend in any mode (tcp or http) as long as they are in the up state
(via tcp-connect or httpchk health checks). In addition haproxy logs the
correct server although it dispatches the request to a wrong server.

I could not reproduce this issue on 2.0.14 or any 2.1.x version. This
happens in tcp and http mode and http requests might be dispatched to
tcp servers and vice versa.

I have tried to narrow this problem down in source using git bisect,
which results in this commit marked as the first bad one:

I have created a setup with a minimal config to reproduce this
unintended behavior with a high probability to occur. The odds of this
bug occuring can be increased by having more backend servers using
health checks. With 2 faulty servers without health checks and 20
servers with health checks I get about a 90-95% chance for a wrong dispatch.

reduced haproxy.cfg:
# note: replace with the internal ip of the host running the
container, i.e. when using
# docker or the container names when using a container-network
# make sure port 8999 is not available
  mode  http
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 5s
  timeout  client 1m
  timeout  server 1m

frontend fe_http_in
  use_backend be_bad.example.com if { req.hdr(host) bad.example.com }
  use_backend be_good.example.com if { req.hdr(host) good.example.com }

backend be_bad.example.com
  server bad.example.com_8999 # make sure this port is
not bound

backend be_good.example.com
  server good.example.com_8070 check

listen li_bad.example.com_tcp_39100:
  mode tcp
  server bad.example.com_tcp_8999 # make sure this port
is not bound

listen li_good.example.com_tcp_39200:
  mode tcp
  server good.example.com_tcp_8071 check

running test-webservices:
podman run -d --rm -p 8070:80 --name nginxdemo nginxdemos/hello
podman run -d --rm -p 8071:8000 --name crccheckdemo crccheck/hello-world
# note: I am running to different webservices to highlight the random
aspect for the redispatch

run haproxy inside a container:
podman run -it --rm \
  --name haproxy \
  -v "${PWD}/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:z" \
  -p 8100:8100 \
  -p 39100:39100 \
  -p 39200:39200 \
# note: I have selinux enabled and thus require :z or :Z to mount a file
or directory into the container

testing using curl:
# expected: HTTP/1.1 503 Service Unavailable
curl -sv -o /dev/null http://bad.example.com --connect-to
:: 2>&1 | grep HTTP/1
# expected: nothing (curl writes "Empty reply from server")
curl -sv -o /dev/null 2>&1 | grep HTTP/1

# expected: HTTP/1.1 200 OK
curl -sv -o /dev/null http://good.example.com --connect-to
:: 2>&1 | grep HTTP/1
# expected: HTTP/1.0 200 OK
curl -sv -o /dev/null 2>&1 | grep HTTP/1

In this setup the curls which get mismatched to the wrong backend server
flip between either HTTP/1.1 when dispatched to the nginxdemos/hello,
between HTTP/1.0 when dispatched to the crccheck/hello-world or the
correct response (503 or nothing) in consecutive runs.

I have attached a simple script which recreates this small test-setup
using podman but it could fairly easily be converted to docker.


Attachment: create-setup.sh
Description: application/shellscript

Reply via email to