Re: HAproxy tproxy problem when try to make transparent proxy
Hello, L. Alberto Giménez ha scritto: Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Like I wrote before, I use ubuntu server 9.10, with kernel 2.6.31 and iptables 1.4.4, so with built-in tproxy support (if I'm not wrong). And I compiled Haproxy by hands with correct parameters I think... lsmod [...] nf_tproxy_core24281 xt_socket, [...] haproxy -vv HA-Proxy version 1.4.2 2010/03/17 Copyright 2000-2010 Willy Tarreau Build options : TARGET = linux26 CPU = i686 CC = gcc CFLAGS = -O2 -march=i686 -g OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 [...] The client can't see directly the backend server. ping -c 1 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. From 192.168.1.2 icmp_seq=1 Destination Host Unreachable --- 192.168.0.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms The backend server can't see the clients directly. ping -c 1 192.168.1.2 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. *From 192.168.1.21 icmp_seq=1 Destination Host Unreachable* (not From 192.168.0.2 like expected) --- 192.168.1.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms So, incredible.. I find the trick.. Alberto, you save my mind.. :-) In backend server I have 2nd ethernet card configured with 192.168.1.21. The cable is out but I forget to disable it (how I'm chicken..).. So everytime the backend try to access to client from this route. Many times errors are in the most simple things. Thanks, thank you very much.. Really! Daniele Hi all, perhaps there is hope for me too ... I have HAProxy running on 192.168.1.101, with this configuration: ... backend test mode http source 0.0.0.0 usesrc clientip server serv1 192.168.4.41 frontend test mode http bind 192.168.1.101:8090 default_backend test the HAProxy is from 1.5-dev6, build with TARGET=linux26 USE_LINUX_TPROXY=1, kernel is 2.6.26-2-amd64 The server is in my vmware player, debian 6 kernel 2.6.32-5-686). By default, player is bridged and dhcp assings it ip 192.168.1.28 (host xp is 192.168.1.62), and default gateway is 192.168.1.2 So manually change ip address on server (eth0, no other cards) to 192.168.4.41 255.255.0.0, and set default gateway to be 192.168.1.101 (the HAProxy machine). On HAProxy I have done routing commands as Daniele did. And I still get 503. What have I done wrong? Is there any known issue if SERVER is in virtual machine? --- posted at http://www.serverphorums.com http://www.serverphorums.com/read.php?10,120994,707061#msg-707061
Re: HAproxy tproxy problem when try to make transparent proxy
On Sat, Mar 20, 2010 at 02:23:29AM +0100, Daniele Genetti wrote: I verify default gw and it seems correct. I also add rules suggested, but nothing change. The error 503 Service Unavailable persist. So, now I try to do this test. 1) Without transparent proxy on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (ok, connection established showed) 2) With transparent proxy activated on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (nothing showed) So, probably there is a problem forwarding.. I'm right? No, you're not watching the same connections. I'm assuming that 192.168.1.20 is your web server and 192.168.1.21 is your haproxy server. In transparent mode, the web server will see the client's IP address as the source, not the haproxy server. So you must use exactly the same grep on both sides. Also, be sure not to test from 127.0.0.1, otherwise it will not work. But what I find strange in your case is that if the connection appears established on the haproxy server, that means that everything is correct, including routing of backwards packets. Otherwise you would see a SYN_SENT state. Anyone maybe have an idea to resolve this issue? Please simplify the test first. Disable health checks on the server. That way we'll know that health checks are not seeing the server as down. Next step is to ensure that you're sending the request from a machine that must be routed back via the haproxy server, so it must not be on the same local net as your web server. If you still don't see any progress, please take a tcpdump capture on both sides (haproxy server and web server). Regards, Willy
Re: HAproxy tproxy problem when try to make transparent proxy
On 03/20/2010 08:27 PM, Daniele Genetti wrote: So, there is something that don't permit to communicate in transparent mode.. Where is the barrier? mmm.. Hi, Sorry for insist on that, but are you *completely* sure that your routing is properly set up so transparent mode can work? This kind of errors are almost always related to routing issues. Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Best regards, L. Alberto Giménez
Re: HAproxy tproxy problem when try to make transparent proxy
Hello, L. Alberto Giménez ha scritto: Please check that: * You have the tproxy enabled in your kernel * You have haproxy compiled with tproxy support Your backend servers *can't* see the clients directly (i.e., they have the haproxy box as default gateway and *no other* gateways). The same for the clients (not mandatory, but if they can see the servers, it may cause trouble). Like I wrote before, I use ubuntu server 9.10, with kernel 2.6.31 and iptables 1.4.4, so with built-in tproxy support (if I'm not wrong). And I compiled Haproxy by hands with correct parameters I think... lsmod [...] nf_tproxy_core24281 xt_socket,[permanent] [...] haproxy -vv HA-Proxy version 1.4.2 2010/03/17 Copyright 2000-2010 Willy Tarreau w...@1wt.eu Build options : TARGET = linux26 CPU = i686 CC = gcc CFLAGS = -O2 -march=i686 -g OPTIONS = USE_LINUX_TPROXY=1 USE_STATIC_PCRE=1 [...] The client can't see directly the backend server. ping -c 1 192.168.0.2 PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data. From 192.168.1.2 icmp_seq=1 Destination Host Unreachable --- 192.168.0.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms The backend server can't see the clients directly. ping -c 1 192.168.1.2 PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data. *From 192.168.1.21 icmp_seq=1 Destination Host Unreachable* (not From 192.168.0.2 like expected) --- 192.168.1.2 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms So, incredible.. I find the trick.. Alberto, you save my mind.. :-) In backend server I have 2nd ethernet card configured with 192.168.1.21. The cable is out but I forget to disable it (how I'm chicken..).. So everytime the backend try to access to client from this route. Many times errors are in the most simple things. Thanks, thank you very much.. Really! Daniele
Re: HAproxy tproxy problem when try to make transparent proxy
Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy
Re: HAproxy tproxy problem when try to make transparent proxy
Also for some reason if you are using the new kernel and the new iptables (as you seem to be) you need to specify the firewall mark on EVERY interface: ip rule add dev eth0 fwmark 111 lookup 100 ip rule add dev eth1 fwmark 111 lookup 100 ip rule add dev eth2 fwmark 111 lookup 100 ip rule add dev eth3 fwmark 111 lookup 100 Not sure why.. On 19 March 2010 18:55, Willy Tarreau w...@1wt.eu wrote: Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/
Re: HAproxy tproxy problem when try to make transparent proxy
I verify default gw and it seems correct. I also add rules suggested, but nothing change. The error 503 Service Unavailable persist. So, now I try to do this test. 1) Without transparent proxy on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (ok, connection established showed) 2) With transparent proxy activated on HAPROXY_SERVER: netstat -ctnup | grep 192.168.1.20:80 (ok, connection established showed) on WEB_SERVER: netstat -ctnup | grep 192.168.1.21:80 (nothing showed) So, probably there is a problem forwarding.. I'm right? Anyone maybe have an idea to resolve this issue? Thanks, Daniele James Little ha scritto: Also for some reason if you are using the new kernel and the new iptables (as you seem to be) you need to specify the firewall mark on EVERY interface: ip rule add dev eth0 fwmark 111 lookup 100 ip rule add dev eth1 fwmark 111 lookup 100 ip rule add dev eth2 fwmark 111 lookup 100 ip rule add dev eth3 fwmark 111 lookup 100 Not sure why.. On 19 March 2010 18:55, Willy Tarreau w...@1wt.eu wrote: Hi, On Fri, Mar 19, 2010 at 07:03:47PM +0100, Daniele Genetti wrote: Hello, I have one big problem with HAproxy compiled with tproxy support. This is the situation... HAPROXY_SERVER os: ubuntu server kernel: 2.6.31 (so with tproxy support) iptables: 1.4.4 (so with tproxy support) ip: 192.168.1.20 WEB_SERVER os: debian kernel: 2.6.26 iptables: 1.4.2 ip: 192.168.1.21 I set up haproxy and with normal rules and configuration all works well! When I try to set the proxy transparent, adding in the configuration the line: source 0.0.0.0 usesrc clientip I have like result all connection 503 Service Unavailable In HAPROXY_SERVER I added this rules: --- iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 --- And also I changed HAPROXY_SERVER sysctrls with: echo 1 /proc/sys/net/ipv4/conf/all/forwarding echo 1 /proc/sys/net/ipv4/conf/all/send_redirects echo 1 /proc/sys/net/ipv4/conf/eth0/send_redirects Where I'm wrong? Have you got any ideas? Thanks! Daniel I suspect that you forgot to change your servers' default gateway to point to the haproxy machine, and that they are responding directly to the client without passing through haproxy. Regards, Willy -- Regards, Malcolm Turnbull. Loadbalancer.org Ltd. Phone: +44 (0)870 443 8779 http://www.loadbalancer.org/