Hi, On Thu, Oct 03, 2019 at 06:40:08PM -0700, Shishir Kumar Yadav wrote: > Hi All, > > We are using Haproxy 1.8.3 and have a peculiar situation. We want to remove > idle client connections but without specifying HAProxy timeout, basically > we want to rely on backend to detect whether client is idle or not because > sometimes backend might not be able to send any data on a connection for > long time for some reasons and we do not want to close connection with > client because of this.
What you're describing is exactly the default behaviour, so you have nothing to do to have this. > Currently backend is able to detect idle client and > kill the connection between HAproxy and backend, but looks like HAProxy is > not closing the connection with client. The connection b/w backend and > HAProxy goes in FIN_WAIT1 state on the backend side but it is still showing > ESTABLISHED on HAProxy side. If you have FIN_WAIT1 on one side, you must have CLOSE_WAIT on the other size, and the FIN_WAIT1 must immediately become FIN_WAIT2 once the other one acknowledges it. If it's not the case, it means that the FIN sent by the server never reached haproxy. This explains why this last one remained ESTABLISHED and why the other one doesn't switch to FIN_WAIT2. I suspect that you're having a firewall, NAT or L4 load balancer between haproxy and your server, with too short timeouts, and that the connection silently expired in the middle, which is why the FIN didn't pass and why this connection cannot be reused. Note that iptables+conntrack running on either of these are also firewalls, so if they are not properly configured (e.g. too short timeout for established connections), you will face such problems. Another possibility could be that you're in fact running in VMs and that it's the hypervisor which runs conntrack with very short timeouts. We've seen this a little bit over the last few years, report by users hosted by incompetent VM providers, who had to switch to a correct one. > Also the connection b/w client and HAProxy > remains in ESTABLISHED state. This situation does not change unless client > is killed. I have also tried 'option nolinger' and it did not help. > > tcp 5462444 0 127.0.0.1:40567 127.0.0.1:8081 > ESTABLISHED 6768/haproxy (Connection b/w Haproxy and backend - Haproxy > side) > tcp6 0 216432 10.60.106.229:80 10.62.185.40:52924 > ESTABLISHED 6768/haproxy (Connection b/w client and HAProxy) > tcp6 0 4109430 127.0.0.1:8081 127.0.0.1:40567 > FIN_WAIT1 - (Connection b/w Haproxy and backend - backend side > ) Ah that's becoming amusing, it's on the loopback that you're losing packets! Thus I'm pretty sure you're having iptables and incorrectly configured timeouts. Have a look inside /proc/sys/net/netfilter/ and you'll probably find nf_conntrack_timeout_established with a low value in it. Willy
