Hello,
On Wed, May 19, 2021 at 01:52:10PM -0700, Peter Scott (US 172D) wrote:
> We have observed some behavior we do not understand with
> haproxy-1.5.18-6.e17.x86_64 and need help determining what is going on and
> how to get the desired behavior.
>
> The behavior: When an HTTP request is initiated from one of the cluster back
> ends, if the load balancer picks that back end to route the request to, it
> times out and then routes it to a different back end. We want it to not
> time out. Requests can come from a back end when a web page runs a program
> that does a curl on an API that happens to be within the cluster.
>
> The setup: We have a load balancer with four back ends on transparent
> proxying (so that Apache logs show the correct client_ip and not that of the
> LB). The back ends and LB are Centos 7.
>
> The evidence: We have run tcpdump on the load balancer and observed the
> following sequence when running a curl from one of the back ends (say, 'X')
> to the LB address:
>
> 1. Back end X ephemeral port A -> load balancer port 80 connection and
> HTTP request
> 2. Load balancer (looking like back end X) ephemeral port B -> back end
> X connection
> 3. No response from back end X (no ACK, no packets at all)
> 4. Repeated attempts to establish connection from different ephemeral ports
> 5. Load balancer gives up
> 6. Load balancer (looking like back end X) -> back end Y port 80
> connection and HTTP request
> 7. Back end Y -> load balancer: HTTP response
> 8. Load balancer -> back end X: HTTP response
>
> If the LB doesn't pick back end X in step 2, we go from step 1 to step 6.
> The timeout we observed was 15 seconds. haproxy.cfg contains:
>
> retries 3
> timeout connect 5s
>
> When we changed the latter to 2s, the delay went to 6 seconds.
>
> Here are the other parts of haproxy.cfg that might be relevant:
>
> defaults
> mode tcp
> option tcplog
> log global
> option tcpka
> option dontlognull
> option http-server-close
> option redispatch
> retries 3
> default-server inter 15s
> timeout http-request 3m
> timeout client 3m
> timeout server 3m
> timeout http-keep-alive 3m
> timeout queue 2m
> timeout connect 5s
> timeout check 10s
>
> frontend internal_http_80
> bind xxx.xxx.xxx.xxx:80 transparent
> default_backend internal_http_80_backend
>
> backend internal_http_80_backend
> balance leastconn
> source 0.0.0.0 usesrc clientip
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You're running in transparent mode, which means that you're spoofing
the client's IP address when connecting to it. You cannot do that to
reconnect to the same node, because the machine definitely sees that
you're using its own IP address to connect, and even if it would
accept it, it would respond to itself!
You need to disable this. Note that in addition, you even have the
same problem if you'd connect to any machine present on the same
network. For example if A connects to LB which connects to B from
A's source address, and B routes the SYN-ACK back directly to A
without passing through the LB, it will fail.
Interestingly, 20 years ago when load balancers used to work at the
network level and were mostly modified routers, this problem was
common and some of them used to implement a source-NAT mechanism on
outgoing traffic called "proxy mode" which made the LB present its
own address when connecting to the servers to avoid this problem.
(...)
> We know the version is quite old and have no problem with upgrading, but
> it's a sufficiently critical environment that we'd like more than a guess
> that the source of the problem is a bug in this version, particularly since
> we've been unable to find any discussion of this behavior through googling.
Based on what you're doing it's no big deal if you keep this version,
you need to fix the configuration and/or the architecture so that under
no circumstance the LB connects to a server pretending to be itself or
pretending to be any address that the server will not route back through
the LB.
Hoping this helps,
Willy
