what does dmesg says then? Or errors logged by HAProxy? You may have some iptables issues or source port exhaustion.
Baptiste On Mon, Jan 26, 2015 at 2:53 PM, Zbyněk Rozman <[email protected]> wrote: > Hi Babtiste, > > yes we do have change that: > > [root@srvA ~]# cat /etc/sysconfig/network-scripts/route-eth0 > default via 192.168.1.254 dev eth0 table 700 > > [root@srvA ~]# cat /etc/sysconfig/network-scripts/rule-eth0 > from 192.168.1.1 lookup 700 > > The setup is working just fine, for a while. When we try to overwhelm the > proxy with thousands of requests from single IP (within same network or > outside, doesnt matter) it gets stuck after a while. Then again turning off > tproxy settings (source 0.0.0.0 usesrc clientip) solves the problem and all > starts working again. > > V Mon, 26 Jan 2015 13:57:24 +0100 > Baptiste <[email protected]> napsáno: > >> Hi Zbynek, >> >> Have you changed the default gateway of your server? >> traffic from server to client must pass through HAProxy box. >> >> In your case, I guess HAProxy sends a SYN to the server and the >> servers sends the S/A to the client directly, bypassing HAProxy. >> >> Baptiste >> >> >> On Mon, Jan 26, 2015 at 1:24 PM, Zbyněk Rozman <[email protected]> >> wrote: >> > Hi, >> > >> > we are currently facing problem with connection clogging with this >> > configuration. >> > The client after a while (random number of requests ~few hundreds) reaches >> > "timeout connect" settings on backend server - proxy returns 503 Service >> > Unavailable No server is available to handle this request. Other clients >> > on different IPs are connecting to server just fine. >> > Strace on haproxy when its not working shows just: Resource temporarily >> > unavailable. >> > After a while it starts working again for random number of requests and >> > then 503 occurs again. >> > When switched to another backend server it starts working again. Switching >> > back to broken backend - still request timeouts for affected clients. >> > It looks like it is only affecting clients behind public IP address with >> > high traffic. >> > We have tried to change various sysctl settings, with no success so far. >> > When we remove the haproxy setting "source 0.0.0.0 usesrc clientip" all >> > works just fine without clogging - so it looks like its tproxy problem, >> > but we are unable to find solution. >> > >> > *** backend server uname -a: >> > Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC >> > 2014 x86_64 x86_64 x86_64 GNU/Linux >> > >> > *** haproxy -vv >> > HA-Proxy version 1.5.8 2014/10/31 >> > Copyright 2000-2014 Willy Tarreau <[email protected]> >> > >> > Build options : >> > TARGET = linux26 >> > CPU = generic >> > CC = gcc >> > CFLAGS = -O2 -g -fno-strict-aliasing >> > OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 >> > >> > Default settings : >> > maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 >> > >> > Encrypted password support via crypt(3): yes >> > Built without zlib support (USE_ZLIB not set) >> > Compression algorithms supported : identity >> > Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 >> > Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 >> > OpenSSL library supports TLS extensions : yes >> > OpenSSL library supports SNI : yes >> > OpenSSL library supports prefer-server-ciphers : yes >> > Built with PCRE version : 7.8 2008-09-05 >> > PCRE library supports JIT : no (USE_PCRE_JIT not set) >> > Built with transparent proxy support using: IP_TRANSPARENT >> > IPV6_TRANSPARENT IP_FREEBIND >> > >> > Available polling systems : >> > epoll : pref=300, test result OK >> > poll : pref=200, test result OK >> > select : pref=150, test result OK >> > Total: 3 (3 usable), will use epoll. >> > >> > *** haproxy server uname -a >> > Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 >> > UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> > >> > *** haproxy server settings: >> > >> > iptables -t mangle -N DIVERT >> > iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT >> > iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT >> > iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT >> > iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT >> > iptables -t mangle -A DIVERT -j MARK --set-mark 1 >> > iptables -t mangle -A DIVERT -j ACCEPT >> > >> > ip rule add fwmark 1 lookup 100 >> > ip route add local 0.0.0.0/0 dev lo table 100 >> > >> > sysctl -w net.ipv4.ip_forward=1 >> > sysctl -w net.ipv4.ip_nonlocal_bind=1 >> > >> > *** haproxy.cfg: >> > >> > frontend front >> > bind IP:80 >> > mode http >> > >> > default_backend WWW >> > capture cookie SERVERID len 32 >> > capture request header Content-Length len 10 >> > reqadd X-Forwarded-Proto:\ http >> > timeout connect 6s >> > >> > backend WWW >> > server srvA 192.168.1.1:80 weight 1 check source 0.0.0.0 >> > usesrc clientip >> > >> > >> > *** sysctl.cfg: >> > >> > kernel.shmmax = 4294967295 >> > kernel.shmall = 268435456 >> > net.ipv4.ip_nonlocal_bind = 1 >> > net.ipv4.ip_forward = 1 >> > net.ipv4.tcp_slow_start_after_idle=0 >> > net.nf_conntrack_max = 200000 >> > net.netfilter.nf_conntrack_tcp_timeout_established = 1800 >> > net.core.rmem_max=16777216 >> > net.core.wmem_max=16777216 >> > net.ipv4.tcp_rmem=4096 87380 16777216 >> > net.ipv4.tcp_wmem=4096 65536 16777216 >> > net.core.netdev_max_backlog = 30000 >> > net.ipv4.tcp_max_syn_backlog = 8192 >> > net.ipv4.ip_local_port_range = 1024 65535 >> > net.core.somaxconn = 40000 >> > net.ipv4.tcp_max_tw_buckets = 1440000 >> > net.ipv4.tcp_synack_retries = 2 >> > net.ipv4.tcp_syn_retries = 2 >> > >> > -- >> > S pozdravem >> > >> > Zbyněk Rozman >> > Linux System Administrator >> > >> > [email protected] >> > 480 020 191 >> > >> > Stable.cz s.r.o. | www.stable.cz >> > Skelná 29, 466 02, Jablonec nad Nisou >> > Tel.: 480 020 191, fax: 480 020 091 >> > >> > napoveda.stable.cz >> > >> > www.stable.cz | www.otoman.cz | www.kvapem.cz | www.multak.cz >> >
