We have observed some behavior we do not understand with
haproxy-1.5.18-6.e17.x86_64 and need help determining what is going on
and how to get the desired behavior.
The behavior: When an HTTP request is initiated from one of the cluster
back ends, if the load balancer picks that back end to route the request
to, it times out and then routes it to a different back end. We want it
to not time out. Requests can come from a back end when a web page runs
a program that does a curl on an API that happens to be within the cluster.
The setup: We have a load balancer with four back ends on transparent
proxying (so that Apache logs show the correct client_ip and not that of
the LB). The back ends and LB are Centos 7.
The evidence: We have run tcpdump on the load balancer and observed the
following sequence when running a curl from one of the back ends (say,
'X') to the LB address:
1. Back end X ephemeral port A -> load balancer port 80 connection and
HTTP request
2. Load balancer (looking like back end X) ephemeral port B -> back end
X connection
3. No response from back end X (no ACK, no packets at all)
4. Repeated attempts to establish connection from different ephemeral ports
5. Load balancer gives up
6. Load balancer (looking like back end X) -> back end Y port 80
connection and HTTP request
7. Back end Y -> load balancer: HTTP response
8. Load balancer -> back end X: HTTP response
If the LB doesn't pick back end X in step 2, we go from step 1 to step
6. The timeout we observed was 15 seconds. haproxy.cfg contains:
retries 3
timeout connect 5s
When we changed the latter to 2s, the delay went to 6 seconds.
Here are the other parts of haproxy.cfg that might be relevant:
defaults
mode tcp
option tcplog
log global
option tcpka
option dontlognull
option http-server-close
option redispatch
retries 3
default-server inter 15s
timeout http-request 3m
timeout client 3m
timeout server 3m
timeout http-keep-alive 3m
timeout queue 2m
timeout connect 5s
timeout check 10s
frontend internal_http_80
bind xxx.xxx.xxx.xxx:80 transparent
default_backend internal_http_80_backend
backend internal_http_80_backend
balance leastconn
source 0.0.0.0 usesrc clientip
option httpchk GET /health.php HTTP/1.0
http-check expect string PASS
server www1 xxx.xxx.xxx.xxx:80 check
server www2 xxx.xxx.xxx.xxx:80 check
server www3 xxx.xxx.xxx.xxx:80 check
server www4 xxx.xxx.xxx.xxx:80 check
The back end IPs in the last block are private RFC 1918 addresses to
avoid sending traffic through unnecessary routers; we don't think that's
related.
We know the version is quite old and have no problem with upgrading, but
it's a sufficiently critical environment that we'd like more than a
guess that the source of the problem is a bug in this version,
particularly since we've been unable to find any discussion of this
behavior through googling.
