On Thu, Oct 03, 2019 at 06:40:08PM -0700, Shishir Kumar Yadav wrote:
> Hi All,
> We are using Haproxy 1.8.3 and have a peculiar situation. We want to remove
> idle client connections but without specifying HAProxy timeout, basically
> we want to rely on backend to detect whether client is idle or not because
> sometimes backend might not be able to send any data on a connection for
> long time for some reasons and we do not want to close connection with
> client because of this.

What you're describing is exactly the default behaviour, so you have
nothing to do to have this.

> Currently backend is able to detect idle client and
> kill the connection between HAproxy and backend, but looks like HAProxy is
> not closing the connection with client. The connection b/w backend and
> HAProxy goes in FIN_WAIT1 state on the backend side but it is still showing
> ESTABLISHED on HAProxy side.

If you have FIN_WAIT1 on one side, you must have CLOSE_WAIT on the other
size, and the FIN_WAIT1 must immediately become FIN_WAIT2 once the other
one acknowledges it. If it's not the case, it means that the FIN sent by
the server never reached haproxy. This explains why this last one remained
ESTABLISHED and why the other one doesn't switch to FIN_WAIT2. I suspect
that you're having a firewall, NAT or L4 load balancer between haproxy
and your server, with too short timeouts, and that the connection silently
expired in the middle, which is why the FIN didn't pass and why this
connection cannot be reused.

Note that iptables+conntrack running on either of these are also firewalls,
so if they are not properly configured (e.g. too short timeout for established
connections), you will face such problems.

Another possibility could be that you're in fact running in VMs and that
it's the hypervisor which runs conntrack with very short timeouts. We've
seen this a little bit over the last few years, report by users hosted
by incompetent VM providers, who had to switch to a correct one.

> Also the connection b/w client and HAProxy
> remains in ESTABLISHED state. This situation does not change unless client
> is killed. I have also tried 'option nolinger' and it did not help.
> tcp   5462444      0
>  ESTABLISHED 6768/haproxy  (Connection b/w Haproxy and backend - Haproxy
> side)
> tcp6       0  216432
>  ESTABLISHED 6768/haproxy  (Connection b/w client and HAProxy)
> tcp6       0 4109430
> FIN_WAIT1   -             (Connection b/w Haproxy and backend - backend side
> )

Ah that's becoming amusing, it's on the loopback that you're losing
packets! Thus I'm pretty sure you're having iptables and incorrectly
configured timeouts. Have a look inside /proc/sys/net/netfilter/ and
you'll probably find nf_conntrack_timeout_established with a low
value in it.


Reply via email to