haproxy + tproxy problem
Hi, we are currently facing problem with connection clogging with this configuration. The client after a while (random number of requests ~few hundreds) reaches timeout connect settings on backend server - proxy returns 503 Service Unavailable No server is available to handle this request. Other clients on different IPs are connecting to server just fine. Strace on haproxy when its not working shows just: Resource temporarily unavailable. After a while it starts working again for random number of requests and then 503 occurs again. When switched to another backend server it starts working again. Switching back to broken backend - still request timeouts for affected clients. It looks like it is only affecting clients behind public IP address with high traffic. We have tried to change various sysctl settings, with no success so far. When we remove the haproxy setting source 0.0.0.0 usesrc clientip all works just fine without clogging - so it looks like its tproxy problem, but we are unable to find solution. *** backend server uname -a: Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy -vv HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. *** haproxy server uname -a Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy server settings: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv4.ip_nonlocal_bind=1 *** haproxy.cfg: frontend front bind IP:80 mode http default_backend WWW capture cookie SERVERID len 32 capture request header Content-Length len 10 reqadd X-Forwarded-Proto:\ http timeout connect 6s backend WWW server srvA192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc clientip *** sysctl.cfg: kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 net.ipv4.tcp_slow_start_after_idle=0 net.nf_conntrack_max = 20 net.netfilter.nf_conntrack_tcp_timeout_established = 1800 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.core.netdev_max_backlog = 3 net.ipv4.tcp_max_syn_backlog = 8192 net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 4 net.ipv4.tcp_max_tw_buckets = 144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 -- S pozdravem Zbyněk Rozman Linux System Administrator zbynek.roz...@stable.cz 480 020 191 Stable.cz s.r.o. | www.stable.cz Skelná 29, 466 02, Jablonec nad Nisou Tel.: 480 020 191, fax: 480 020 091 napoveda.stable.cz www.stable.cz | www.otoman.cz | www.kvapem.cz | www.multak.cz
Re: haproxy + tproxy problem
Hi Babtiste, yes we do have change that: [root@srvA ~]# cat /etc/sysconfig/network-scripts/route-eth0 default via 192.168.1.254 dev eth0 table 700 [root@srvA ~]# cat /etc/sysconfig/network-scripts/rule-eth0 from 192.168.1.1 lookup 700 The setup is working just fine, for a while. When we try to overwhelm the proxy with thousands of requests from single IP (within same network or outside, doesnt matter) it gets stuck after a while. Then again turning off tproxy settings (source 0.0.0.0 usesrc clientip) solves the problem and all starts working again. V Mon, 26 Jan 2015 13:57:24 +0100 Baptiste bed...@gmail.com napsáno: Hi Zbynek, Have you changed the default gateway of your server? traffic from server to client must pass through HAProxy box. In your case, I guess HAProxy sends a SYN to the server and the servers sends the S/A to the client directly, bypassing HAProxy. Baptiste On Mon, Jan 26, 2015 at 1:24 PM, Zbyněk Rozman zbynek.roz...@stable.cz wrote: Hi, we are currently facing problem with connection clogging with this configuration. The client after a while (random number of requests ~few hundreds) reaches timeout connect settings on backend server - proxy returns 503 Service Unavailable No server is available to handle this request. Other clients on different IPs are connecting to server just fine. Strace on haproxy when its not working shows just: Resource temporarily unavailable. After a while it starts working again for random number of requests and then 503 occurs again. When switched to another backend server it starts working again. Switching back to broken backend - still request timeouts for affected clients. It looks like it is only affecting clients behind public IP address with high traffic. We have tried to change various sysctl settings, with no success so far. When we remove the haproxy setting source 0.0.0.0 usesrc clientip all works just fine without clogging - so it looks like its tproxy problem, but we are unable to find solution. *** backend server uname -a: Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy -vv HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. *** haproxy server uname -a Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy server settings: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv4.ip_nonlocal_bind=1 *** haproxy.cfg: frontend front bind IP:80 mode http default_backend WWW capture cookie SERVERID len 32 capture request header Content-Length len 10 reqadd X-Forwarded-Proto:\ http timeout connect 6s backend WWW server srvA192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc clientip *** sysctl.cfg: kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 net.ipv4.tcp_slow_start_after_idle=0 net.nf_conntrack_max = 20 net.netfilter.nf_conntrack_tcp_timeout_established = 1800 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.core.netdev_max_backlog
Re: haproxy + tproxy problem
Hi Zbynek, Have you changed the default gateway of your server? traffic from server to client must pass through HAProxy box. In your case, I guess HAProxy sends a SYN to the server and the servers sends the S/A to the client directly, bypassing HAProxy. Baptiste On Mon, Jan 26, 2015 at 1:24 PM, Zbyněk Rozman zbynek.roz...@stable.cz wrote: Hi, we are currently facing problem with connection clogging with this configuration. The client after a while (random number of requests ~few hundreds) reaches timeout connect settings on backend server - proxy returns 503 Service Unavailable No server is available to handle this request. Other clients on different IPs are connecting to server just fine. Strace on haproxy when its not working shows just: Resource temporarily unavailable. After a while it starts working again for random number of requests and then 503 occurs again. When switched to another backend server it starts working again. Switching back to broken backend - still request timeouts for affected clients. It looks like it is only affecting clients behind public IP address with high traffic. We have tried to change various sysctl settings, with no success so far. When we remove the haproxy setting source 0.0.0.0 usesrc clientip all works just fine without clogging - so it looks like its tproxy problem, but we are unable to find solution. *** backend server uname -a: Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy -vv HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. *** haproxy server uname -a Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy server settings: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv4.ip_nonlocal_bind=1 *** haproxy.cfg: frontend front bind IP:80 mode http default_backend WWW capture cookie SERVERID len 32 capture request header Content-Length len 10 reqadd X-Forwarded-Proto:\ http timeout connect 6s backend WWW server srvA192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc clientip *** sysctl.cfg: kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 net.ipv4.tcp_slow_start_after_idle=0 net.nf_conntrack_max = 20 net.netfilter.nf_conntrack_tcp_timeout_established = 1800 net.core.rmem_max=16777216 net.core.wmem_max=16777216 net.ipv4.tcp_rmem=4096 87380 16777216 net.ipv4.tcp_wmem=4096 65536 16777216 net.core.netdev_max_backlog = 3 net.ipv4.tcp_max_syn_backlog = 8192 net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 4 net.ipv4.tcp_max_tw_buckets = 144 net.ipv4.tcp_synack_retries = 2 net.ipv4.tcp_syn_retries = 2 -- S pozdravem Zbyněk Rozman Linux System Administrator zbynek.roz...@stable.cz 480 020 191 Stable.cz s.r.o. | www.stable.cz Skelná 29, 466 02, Jablonec nad Nisou Tel.: 480 020 191, fax: 480 020 091 napoveda.stable.cz www.stable.cz | www.otoman.cz | www.kvapem.cz | www.multak.cz
Re: haproxy + tproxy problem
what does dmesg says then? Or errors logged by HAProxy? You may have some iptables issues or source port exhaustion. Baptiste On Mon, Jan 26, 2015 at 2:53 PM, Zbyněk Rozman zbynek.roz...@stable.cz wrote: Hi Babtiste, yes we do have change that: [root@srvA ~]# cat /etc/sysconfig/network-scripts/route-eth0 default via 192.168.1.254 dev eth0 table 700 [root@srvA ~]# cat /etc/sysconfig/network-scripts/rule-eth0 from 192.168.1.1 lookup 700 The setup is working just fine, for a while. When we try to overwhelm the proxy with thousands of requests from single IP (within same network or outside, doesnt matter) it gets stuck after a while. Then again turning off tproxy settings (source 0.0.0.0 usesrc clientip) solves the problem and all starts working again. V Mon, 26 Jan 2015 13:57:24 +0100 Baptiste bed...@gmail.com napsáno: Hi Zbynek, Have you changed the default gateway of your server? traffic from server to client must pass through HAProxy box. In your case, I guess HAProxy sends a SYN to the server and the servers sends the S/A to the client directly, bypassing HAProxy. Baptiste On Mon, Jan 26, 2015 at 1:24 PM, Zbyněk Rozman zbynek.roz...@stable.cz wrote: Hi, we are currently facing problem with connection clogging with this configuration. The client after a while (random number of requests ~few hundreds) reaches timeout connect settings on backend server - proxy returns 503 Service Unavailable No server is available to handle this request. Other clients on different IPs are connecting to server just fine. Strace on haproxy when its not working shows just: Resource temporarily unavailable. After a while it starts working again for random number of requests and then 503 occurs again. When switched to another backend server it starts working again. Switching back to broken backend - still request timeouts for affected clients. It looks like it is only affecting clients behind public IP address with high traffic. We have tried to change various sysctl settings, with no success so far. When we remove the haproxy setting source 0.0.0.0 usesrc clientip all works just fine without clogging - so it looks like its tproxy problem, but we are unable to find solution. *** backend server uname -a: Linux backend 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy -vv HA-Proxy version 1.5.8 2014/10/31 Copyright 2000-2014 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. *** haproxy server uname -a Linux haproxy-test 2.6.32-504.3.3.el6.x86_64 #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux *** haproxy server settings: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 80 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 81 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 82 -m socket -j DIVERT iptables -t mangle -A PREROUTING -p tcp --sport 443 -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv4.ip_nonlocal_bind=1 *** haproxy.cfg: frontend front bind IP:80 mode http default_backend WWW capture cookie SERVERID len 32 capture request header Content-Length len 10 reqadd X-Forwarded-Proto:\ http timeout connect 6s backend WWW server srvA192.168.1.1:80 weight 1 check source 0.0.0.0 usesrc clientip *** sysctl.cfg: kernel.shmmax = 4294967295 kernel.shmall = 268435456 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 net.ipv4.tcp_slow_start_after_idle=0 net.nf_conntrack_max = 20
Re: HAproxy tproxy problem when try to make transparent proxy
Hello, L. Alberto Giménez ha scritto: Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Like I wrote before, I use ubuntu server 9.10, with kernel 2.6.31 and iptables 1.4.4, so with built-in tproxy support (if I'm not wrong). And I compiled Haproxy by hands with correct parameters I think... lsmod [...] nf_tproxy_core24281 xt_socket, [...] haproxy -vv HA-Proxy version 1.4.2 2010/03/17 Copyright 2000-2010 Willy Tarreau Build options : TARGET = linux26 CPU = i686 CC = gcc CFLAGS = -O2 -march=i686 -g OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 [...] The client can't see directly the backend server. ping -c 1 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. From 192.168.1.2 icmp_seq=1 Destination Host Unreachable --- 192.168.0.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms The backend server can't see the clients directly. ping -c 1 192.168.1.2 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. *From 192.168.1.21 icmp_seq=1 Destination Host Unreachable* (not From 192.168.0.2 like expected) --- 192.168.1.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms So, incredible.. I find the trick.. Alberto, you save my mind.. :-) In backend server I have 2nd ethernet card configured with 192.168.1.21. The cable is out but I forget to disable it (how I'm chicken..).. So everytime the backend try to access to client from this route. Many times errors are in the most simple things. Thanks, thank you very much.. Really! Daniele Hi all, perhaps there is hope for me too ... I have HAProxy running on 192.168.1.101, with this configuration: ... backend test mode http source 0.0.0.0 usesrc clientip server serv1 192.168.4.41 frontend test mode http bind 192.168.1.101:8090 default_backend test the HAProxy is from 1.5-dev6, build with TARGET=linux26 USE_LINUX_TPROXY=1, kernel is 2.6.26-2-amd64 The server is in my vmware player, debian 6 kernel 2.6.32-5-686). By default, player is bridged and dhcp assings it ip 192.168.1.28 (host xp is 192.168.1.62), and default gateway is 192.168.1.2 So manually change ip address on server (eth0, no other cards) to 192.168.4.41 255.255.0.0, and set default gateway to be 192.168.1.101 (the HAProxy machine). On HAProxy I have done routing commands as Daniele did. And I still get 503. What have I done wrong? Is there any known issue if SERVER is in virtual machine? --- posted at http://www.serverphorums.com http://www.serverphorums.com/read.php?10,120994,707061#msg-707061
Re: HAproxy tproxy problem when try to make transparent proxy
On Sat, Mar 20, 2010 at 02:23:29AM +0100, Daniele Genetti wrote: I verify default gw and it seems correct. I also add rules suggested, but nothing change. The error 503 Service Unavailable persist. So, now I try to do this test. 1) Without transparent proxy on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (ok, connection established showed) 2) With transparent proxy activated on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (nothing showed) So, probably there is a problem forwarding.. I'm right? No, you're not watching the same connections. I'm assuming that 192.168.1.20 is your web server and 192.168.1.21 is your haproxy server. In transparent mode, the web server will see the client's IP address as the source, not the haproxy server. So you must use exactly the same grep on both sides. Also, be sure not to test from 127.0.0.1, otherwise it will not work. But what I find strange in your case is that if the connection appears established on the haproxy server, that means that everything is correct, including routing of backwards packets. Otherwise you would see a SYN_SENT state. Anyone maybe have an idea to resolve this issue? Please simplify the test first. Disable health checks on the server. That way we'll know that health checks are not seeing the server as down. Next step is to ensure that you're sending the request from a machine that must be routed back via the haproxy server, so it must not be on the same local net as your web server. If you still don't see any progress, please take a tcpdump capture on both sides (haproxy server and web server). Regards, Willy
Re: HAproxy tproxy problem when try to make transparent proxy
On 03/20/2010 08:27 PM, Daniele Genetti wrote: So, there is something that don't permit to communicate in transparent mode.. Where is the barrier? mmm.. Hi, Sorry for insist on that, but are you *completely* sure that your routing is properly set up so transparent mode can work? This kind of errors are almost always related to routing issues. Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Best regards, L. Alberto Giménez
Re: HAproxy tproxy problem when try to make transparent proxy
Hello, L. Alberto Giménez ha scritto: Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Like I wrote before, I use ubuntu server 9.10, with kernel 2.6.31 and iptables 1.4.4, so with built-in tproxy support (if I'm not wrong). And I compiled Haproxy by hands with correct parameters I think... lsmod [...] nf_tproxy_core24281 xt_socket,[permanent] [...] haproxy -vv HA-Proxy version 1.4.2 2010/03/17 Copyright 2000-2010 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = i686 CC = gcc CFLAGS = -O2 -march=i686 -g OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 [...] The client can't see directly the backend server. ping -c 1 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. From 192.168.1.2 icmp_seq=1 Destination Host Unreachable --- 192.168.0.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms The backend server can't see the clients directly. ping -c 1 192.168.1.2 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. *From 192.168.1.21 icmp_seq=1 Destination Host Unreachable* (not From 192.168.0.2 like expected) --- 192.168.1.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms So, incredible.. I find the trick.. Alberto, you save my mind.. :-) In backend server I have 2nd ethernet card configured with 192.168.1.21. The cable is out but I forget to disable it (how I'm chicken..).. So everytime the backend try to access to client from this route. Many times errors are in the most simple things. Thanks, thank you very much.. Really! Daniele
Re: HAproxy tproxy problem when try to make transparent proxy
Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy
Re: HAproxy tproxy problem when try to make transparent proxy
Also for some reason if you are using the new kernel and the new iptables (as you seem to be) you need to specify the firewall mark on EVERY interface: ip rule add dev eth0 fwmark 111 lookup 100 ip rule add dev eth1 fwmark 111 lookup 100 ip rule add dev eth2 fwmark 111 lookup 100 ip rule add dev eth3 fwmark 111 lookup 100 Not sure why.. On 19 March 2010 18:55, Willy Tarreau w...@1wt.eu wrote: Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: HAproxy tproxy problem when try to make transparent proxy
I verify default gw and it seems correct. I also add rules suggested, but nothing change. The error 503 Service Unavailable persist. So, now I try to do this test. 1) Without transparent proxy on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (ok, connection established showed) 2) With transparent proxy activated on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (nothing showed) So, probably there is a problem forwarding.. I'm right? Anyone maybe have an idea to resolve this issue? Thanks, Daniele James Little ha scritto: Also for some reason if you are using the new kernel and the new iptables (as you seem to be) you need to specify the firewall mark on EVERY interface: ip rule add dev eth0 fwmark 111 lookup 100 ip rule add dev eth1 fwmark 111 lookup 100 ip rule add dev eth2 fwmark 111 lookup 100 ip rule add dev eth3 fwmark 111 lookup 100 Not sure why.. On 19 March 2010 18:55, Willy Tarreau w...@1wt.eu wrote: Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
