Thanks. Inline:
On 5/19/2021 8:23 PM, Willy Tarreau wrote:
Hello,
On Wed, May 19, 2021 at 01:52:10PM -0700, Peter Scott (US 172D) wrote:
We have observed some behavior we do not understand with
haproxy-1.5.18-6.e17.x86_64 and need help determining what is going on and
how to get the desired behavior.
The behavior: When an HTTP request is initiated from one of the cluster back
ends, if the load balancer picks that back end to route the request to, it
times out and then routes it to a different back end. We want it to not
time out. Requests can come from a back end when a web page runs a program
that does a curl on an API that happens to be within the cluster.
The setup: We have a load balancer with four back ends on transparent
proxying (so that Apache logs show the correct client_ip and not that of the
LB). The back ends and LB are Centos 7.
The evidence: We have run tcpdump on the load balancer and observed the
following sequence when running a curl from one of the back ends (say, 'X')
to the LB address:
1. Back end X ephemeral port A -> load balancer port 80 connection and
HTTP request
2. Load balancer (looking like back end X) ephemeral port B -> back end
X connection
3. No response from back end X (no ACK, no packets at all)
4. Repeated attempts to establish connection from different ephemeral ports
5. Load balancer gives up
6. Load balancer (looking like back end X) -> back end Y port 80
connection and HTTP request
7. Back end Y -> load balancer: HTTP response
8. Load balancer -> back end X: HTTP response
If the LB doesn't pick back end X in step 2, we go from step 1 to step 6.
The timeout we observed was 15 seconds. haproxy.cfg contains:
retries 3
timeout connect 5s
When we changed the latter to 2s, the delay went to 6 seconds.
Here are the other parts of haproxy.cfg that might be relevant:
defaults
mode tcp
option tcplog
log global
option tcpka
option dontlognull
option http-server-close
option redispatch
retries 3
default-server inter 15s
timeout http-request 3m
timeout client 3m
timeout server 3m
timeout http-keep-alive 3m
timeout queue 2m
timeout connect 5s
timeout check 10s
frontend internal_http_80
bind xxx.xxx.xxx.xxx:80 transparent
default_backend internal_http_80_backend
backend internal_http_80_backend
balance leastconn
source 0.0.0.0 usesrc clientip
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You're running in transparent mode, which means that you're spoofing
the client's IP address when connecting to it. You cannot do that to
reconnect to the same node, because the machine definitely sees that
you're using its own IP address to connect, and even if it would
accept it, it would respond to itself!
You need to disable this.
And indeed that was how we had originally configured haproxy, but we
switched to transparent mode because we need the correct client IP
logged on the Apache back ends. We couldn't use send-proxy or
forwardfor because haproxy is not doing SSL termination. We couldn't do
SSL termination because the back ends are serving hundreds of virtual
hosts each with their own SSL certs and the certs change without notice
daily as CNAMEs come and go.
We tried send-proxy-v2-ssl and got SSL connection failure. Is that a
valid approach and we just didn't get it right? Otherwise we're reduced
to asking whether there is a way to ensure a connection is not routed to
a particular back end if the connection came from there.
Note that in addition, you even have the
same problem if you'd connect to any machine present on the same
network. For example if A connects to LB which connects to B from
A's source address, and B routes the SYN-ACK back directly to A
without passing through the LB, it will fail.
Interestingly, 20 years ago when load balancers used to work at the
network level and were mostly modified routers, this problem was
common and some of them used to implement a source-NAT mechanism on
outgoing traffic called "proxy mode" which made the LB present its
own address when connecting to the servers to avoid this problem.
(...)
We know the version is quite old and have no problem with upgrading, but
it's a sufficiently critical environment that we'd like more than a guess
that the source of the problem is a bug in this version, particularly since
we've been unable to find any discussion of this behavior through googling.
Based on what you're doing it's no big deal if you keep this version,
you need to fix the configuration and/or the architecture so that under
no circumstance the LB connects to a server pretending to be itself or
pretending to be any address that the server will not route back through
the LB.
Hoping this helps,
Willy
