Hi, we are currently facing problem with connection clogging with this configuration. The client after a while (random number of requests ~few hundreds) reaches "timeout connect" settings on backend server - proxy returns 503 Service Unavailable No server is available to handle this request. Other clients on different IPs are connecting to server just fine. Strace on haproxy when its not working shows just: Resource temporarily unavailable. After a while it starts working again for random number of requests and then 503 occurs again. When switched to another backend server it starts working again. Switching back to broken backend - still request timeouts for affected clients. It looks like it is only affecting clients behind public IP address with high traffic. We have tried to change various sysctl settings, with no success so far. When we remove the haproxy setting "source 0.0.0.0 usesrc clientip" all works just fine without clogging - so it looks like its tproxy problem, but we are unable to find solution.
*** backend server uname -a: Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy -vv HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau <[email protected]> Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. *** haproxy server uname -a Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy server settings: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv4.ip_nonlocal_bind=1 *** haproxy.cfg: frontend front bind IP:80 mode http default_backend WWW capture cookie SERVERID len 32 capture request header Content-Length len 10 reqadd X-Forwarded-Proto:\ http timeout connect 6s backend WWW server srvA 192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc clientip *** sysctl.cfg: kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 net.ipv4.tcp_slow_start_after_idle=0 net.nf_conntrack_max = 200000 net.netfilter.nf_conntrack_tcp_timeout_established = 1800 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.core.netdev_max_backlog = 30000 net.ipv4.tcp_max_syn_backlog = 8192 net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 40000 net.ipv4.tcp_max_tw_buckets = 1440000 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 -- S pozdravem Zbyněk Rozman Linux System Administrator [email protected] 480 020 191 Stable.cz s.r.o. | www.stable.cz Skelná 29, 466 02, Jablonec nad Nisou Tel.: 480 020 191, fax: 480 020 091 napoveda.stable.cz www.stable.cz | www.otoman.cz | www.kvapem.cz | www.multak.cz
