Hi Zbynek, Have you changed the default gateway of your server? traffic from server to client must pass through HAProxy box.
In your case, I guess HAProxy sends a SYN to the server and the servers sends the S/A to the client directly, bypassing HAProxy. Baptiste On Mon, Jan 26, 2015 at 1:24 PM, Zbyněk Rozman <zbynek.roz...@stable.cz> wrote: > Hi, > > we are currently facing problem with connection clogging with this > configuration. > The client after a while (random number of requests ~few hundreds) reaches > "timeout connect" settings on backend server - proxy returns 503 Service > Unavailable No server is available to handle this request. Other clients on > different IPs are connecting to server just fine. > Strace on haproxy when its not working shows just: Resource temporarily > unavailable. > After a while it starts working again for random number of requests and then > 503 occurs again. > When switched to another backend server it starts working again. Switching > back to broken backend - still request timeouts for affected clients. > It looks like it is only affecting clients behind public IP address with high > traffic. > We have tried to change various sysctl settings, with no success so far. > When we remove the haproxy setting "source 0.0.0.0 usesrc clientip" all works > just fine without clogging - so it looks like its tproxy problem, but we are > unable to find solution. > > *** backend server uname -a: > Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux > > *** haproxy -vv > HA-Proxy version 1.5.8 2014/10/31 > Copyright 2000-2014 Willy Tarreau <w...@1wt.eu> > > Build options : > TARGET = linux26 > CPU = generic > CC = gcc > CFLAGS = -O2 -g -fno-strict-aliasing > OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 > > Default settings : > maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 > > Encrypted password support via crypt(3): yes > Built without zlib support (USE_ZLIB not set) > Compression algorithms supported : identity > Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 > Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 > OpenSSL library supports TLS extensions : yes > OpenSSL library supports SNI : yes > OpenSSL library supports prefer-server-ciphers : yes > Built with PCRE version : 7.8 2008-09-05 > PCRE library supports JIT : no (USE_PCRE_JIT not set) > Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT > IP_FREEBIND > > Available polling systems : > epoll : pref=300, test result OK > poll : pref=200, test result OK > select : pref=150, test result OK > Total: 3 (3 usable), will use epoll. > > *** haproxy server uname -a > Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC > 2014 x86_64 x86_64 x86_64 GNU/Linux > > *** haproxy server settings: > > iptables -t mangle -N DIVERT > iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT > iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT > iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT > iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT > iptables -t mangle -A DIVERT -j MARK --set-mark 1 > iptables -t mangle -A DIVERT -j ACCEPT > > ip rule add fwmark 1 lookup 100 > ip route add local 0.0.0.0/0 dev lo table 100 > > sysctl -w net.ipv4.ip_forward=1 > sysctl -w net.ipv4.ip_nonlocal_bind=1 > > *** haproxy.cfg: > > frontend front > bind IP:80 > mode http > > default_backend WWW > capture cookie SERVERID len 32 > capture request header Content-Length len 10 > reqadd X-Forwarded-Proto:\ http > timeout connect 6s > > backend WWW > server srvA 192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc > clientip > > > *** sysctl.cfg: > > kernel.shmmax = 4294967295 > kernel.shmall = 268435456 > net.ipv4.ip_nonlocal_bind = 1 > net.ipv4.ip_forward = 1 > net.ipv4.tcp_slow_start_after_idle=0 > net.nf_conntrack_max = 200000 > net.netfilter.nf_conntrack_tcp_timeout_established = 1800 > net.core.rmem_max=16777216 > net.core.wmem_max=16777216 > net.ipv4.tcp_rmem=4096 87380 16777216 > net.ipv4.tcp_wmem=4096 65536 16777216 > net.core.netdev_max_backlog = 30000 > net.ipv4.tcp_max_syn_backlog = 8192 > net.ipv4.ip_local_port_range = 1024 65535 > net.core.somaxconn = 40000 > net.ipv4.tcp_max_tw_buckets = 1440000 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.tcp_syn_retries = 2 > > -- > S pozdravem > > Zbyněk Rozman > Linux System Administrator > > zbynek.roz...@stable.cz > 480 020 191 > > Stable.cz s.r.o. | www.stable.cz > Skelná 29, 466 02, Jablonec nad Nisou > Tel.: 480 020 191, fax: 480 020 091 > > napoveda.stable.cz > > www.stable.cz | www.otoman.cz | www.kvapem.cz | www.multak.cz >
