Re: Tproxy with multiple interfaces
Hi Brian, > you would then need to setup ipmasq in iptables to make the haproxy server properly route the packets out to the internet and masq the source IP as the haproxy eth0 IP Unfortunately this wouldn't work, as packets route in to the web servers through another router, and would route out through the HAproxy box. I can setup a SNAT rule on the HAproxy box to change the source address, which handles the outbound traffic from the web servers: iptables -t nat -A POSTROUTING -s 192.168.3.65 -j SNAT --to xxx.xxx.97.154 They can route out externally through the HAProxy's eth1 using this method, but it results in the web server being inaccessible externally. The web server would receive a connection on xxx.xxx.97.154 through the router, and send its reply through the HAProxy server. There is probably a way to ensure that packets received on eth0 are routed out through eth0, and likewise for eth1, by using routing tables on the web servers. The major issue with this though, is that the web servers will be running Windows, and won't be capable of this. The only solution I can see to this is to not use the private interfaces altogether, and proxy to the web server's eth0 interfaces. I also can't speak for how the other environment works - it just does. And flawlessly :) Thanks, REW On Tue, Apr 12, 2011 at 5:02 PM, Brian Carpio wrote: > Randy, > > > > I can’t speak to how your other environment works, as it seems suspicious > that it works the way you describe in fully transparent mode but I also > can’t speak to the cttproxy patch as I’ve never used it. When you set the > default gateway on the webservers to the haproxy eth1 interface you would > then need to setup ipmasq in iptables to make the haproxy server properly > route the packets out to the internet and masq the source IP as the haproxy > eth0 IP. > > > > If the other environment is truly working then possibly you need to check > two other settings on the non-working environment. On the haproxy > environment make sure the below are set to 1… Possibly this will resolve > your problems, if it does can you let me know because I can’t seem to wrap > my head around the fact that it would work. > > > > cat /proc/sys/net/ipv4/conf/all/send_redirects > > cat /proc/sys/net/ipv4/conf/eth0/send_redirects > > cat /proc/sys/net/ipv4/conf/eth1/send_redirects > > > > > > *Brian Carpio * > > *Senior Systems Engineer* > > *[image: Description: Description: BroadHop Home Page]* > > *Office: +1.303.962.7242* > > *Mobile: +1.720.319.8617* > > *Email: bcar...@broadhop.com* > > > > *From:* Randy Wilson [mailto:randyedwil...@gmail.com] > *Sent:* Tuesday, April 12, 2011 9:40 AM > *To:* haproxy@formilux.org > *Subject:* Re: Tproxy with multiple interfaces > > > > Hi Brian, > > Thanks for the response. > > I had previously tried this, but setting the default gateway on the web > servers to point to the HAProxy server's eth1 results in the web servers > losing all external connectivity, as the source address is always a private > address. > > root@web:~# ping -c 5 8.8.8.8 > PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. > ^C > --- 8.8.8.8 ping statistics --- > 5 packets transmitted, 0 received, 100% packet loss, time 4008ms > > root@haproxy:~# tcpdump -n -i eth1 icmp > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes > 16:33:57.024999 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq > 1, length 64 > 16:33:58.034080 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq > 2, length 64 > 16:33:59.034036 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq > 3, length 64 > 16:34:00.034104 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq > 4, length 64 > 16:34:01.034266 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq > 5, length 64 > > Any other ideas? > > I'm currently running a similar setup to load balance a mail cluster that's > been in place for almost 4 years. The HAProxy servers use an older kernel > with the cttproxy patch. The mail servers all receive connections from the > HAProxy boxes on their eth1 interfaces and route back out to the them on > their eth0s - without any iptables rules or routing tables. > > > Thanks, > > REW > > > On Tue, Apr 12, 2011 at 4:24 PM, Brian Carpio > wrote: > > Randy, > > The problem is the gateway on the backend webservers needs to be set as a > VIP (or eth1 interface) on the HAproxy servers on their private interface > (assuming you have two HAproxy servers and are using heartbeat for > failover). It looks like from your ro
RE: Tproxy with multiple interfaces
Randy, I can't speak to how your other environment works, as it seems suspicious that it works the way you describe in fully transparent mode but I also can't speak to the cttproxy patch as I've never used it. When you set the default gateway on the webservers to the haproxy eth1 interface you would then need to setup ipmasq in iptables to make the haproxy server properly route the packets out to the internet and masq the source IP as the haproxy eth0 IP. If the other environment is truly working then possibly you need to check two other settings on the non-working environment. On the haproxy environment make sure the below are set to 1... Possibly this will resolve your problems, if it does can you let me know because I can't seem to wrap my head around the fact that it would work. cat /proc/sys/net/ipv4/conf/all/send_redirects cat /proc/sys/net/ipv4/conf/eth0/send_redirects cat /proc/sys/net/ipv4/conf/eth1/send_redirects Brian Carpio Senior Systems Engineer [cid:image001.jpg@01CBF8F7.EABDD0F0] Office: +1.303.962.7242 Mobile: +1.720.319.8617 Email: bcar...@broadhop.com From: Randy Wilson [mailto:randyedwil...@gmail.com] Sent: Tuesday, April 12, 2011 9:40 AM To: haproxy@formilux.org Subject: Re: Tproxy with multiple interfaces Hi Brian, Thanks for the response. I had previously tried this, but setting the default gateway on the web servers to point to the HAProxy server's eth1 results in the web servers losing all external connectivity, as the source address is always a private address. root@web:~# ping -c 5 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ^C --- 8.8.8.8 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4008ms root@haproxy:~# tcpdump -n -i eth1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes 16:33:57.024999 IP 192.168.3.65 > 8.8.8.8<http://8.8.8.8>: ICMP echo request, id 6180, seq 1, length 64 16:33:58.034080 IP 192.168.3.65 > 8.8.8.8<http://8.8.8.8>: ICMP echo request, id 6180, seq 2, length 64 16:33:59.034036 IP 192.168.3.65 > 8.8.8.8<http://8.8.8.8>: ICMP echo request, id 6180, seq 3, length 64 16:34:00.034104 IP 192.168.3.65 > 8.8.8.8<http://8.8.8.8>: ICMP echo request, id 6180, seq 4, length 64 16:34:01.034266 IP 192.168.3.65 > 8.8.8.8<http://8.8.8.8>: ICMP echo request, id 6180, seq 5, length 64 Any other ideas? I'm currently running a similar setup to load balance a mail cluster that's been in place for almost 4 years. The HAProxy servers use an older kernel with the cttproxy patch. The mail servers all receive connections from the HAProxy boxes on their eth1 interfaces and route back out to the them on their eth0s - without any iptables rules or routing tables. Thanks, REW On Tue, Apr 12, 2011 at 4:24 PM, Brian Carpio mailto:bcar...@broadhop.com>> wrote: Randy, The problem is the gateway on the backend webservers needs to be set as a VIP (or eth1 interface) on the HAproxy servers on their private interface (assuming you have two HAproxy servers and are using heartbeat for failover). It looks like from your routing table that eth0 on the webservers' gateway is pointed to the eth0 interface on haproxy this is why it works perfectly when you configure haproxy to use the public IPs on the webservers. Once you change the default gateway on the backend webserver's to use eth1 on the haproxy server (or a VIP which lives on eth1 using heartbeat for failover between two haproxy servers) then it will work. Brian Carpio Senior Systems Engineer Office: +1.303.962.7242 Mobile: +1.720.319.8617 Email: bcar...@broadhop.com<http://bcar...@broadhop.com> From: Randy Wilson [mailto:randyedwil...@gmail.com<mailto:randyedwil...@gmail.com>] Sent: Tuesday, April 12, 2011 8:29 AM To: haproxy@formilux.org<mailto:haproxy@formilux.org> Subject: Tproxy with multiple interfaces Hi, I'm trying to setup an HAProxy instance to transparently load balance a group of web servers. The HAProxy server and web servers each have two interfaces; eth0 as the public interface and eth1 the private. I'm trying to configure the load balancer to accept requests on port 80 on eth0 and transparently proxy the connections to the web servers over the private interfaces on eth1. I've configured the load balancer in the normal way for tproxy, and have the web servers routing out through it. i.e. iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 dev eth0 lookup 100 ip rule add fwmark 1 dev eth1 lookup 100 ip route add local 0.0.0.0/0<http://0.0.0.0/0> dev lo table 100 root@haproxy:~# sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 root@web:~#
Re: Tproxy with multiple interfaces
Hi Brian, Thanks for the response. I had previously tried this, but setting the default gateway on the web servers to point to the HAProxy server's eth1 results in the web servers losing all external connectivity, as the source address is always a private address. root@web:~# ping -c 5 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. ^C --- 8.8.8.8 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 4008ms root@haproxy:~# tcpdump -n -i eth1 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes 16:33:57.024999 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq 1, length 64 16:33:58.034080 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq 2, length 64 16:33:59.034036 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq 3, length 64 16:34:00.034104 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq 4, length 64 16:34:01.034266 IP 192.168.3.65 > 8.8.8.8: ICMP echo request, id 6180, seq 5, length 64 Any other ideas? I'm currently running a similar setup to load balance a mail cluster that's been in place for almost 4 years. The HAProxy servers use an older kernel with the cttproxy patch. The mail servers all receive connections from the HAProxy boxes on their eth1 interfaces and route back out to the them on their eth0s - without any iptables rules or routing tables. Thanks, REW On Tue, Apr 12, 2011 at 4:24 PM, Brian Carpio wrote: > Randy, > > The problem is the gateway on the backend webservers needs to be set as a > VIP (or eth1 interface) on the HAproxy servers on their private interface > (assuming you have two HAproxy servers and are using heartbeat for > failover). It looks like from your routing table that eth0 on the > webservers’ gateway is pointed to the eth0 interface on haproxy this is why > it works perfectly when you configure haproxy to use the public IPs on the > webservers. > > > > Once you change the default gateway on the backend webserver’s to use eth1 > on the haproxy server (or a VIP which lives on eth1 using heartbeat for > failover between two haproxy servers) then it will work. > > > > *Brian Carpio * > > *Senior Systems Engineer* > > *[image: Description: Description: BroadHop Home Page]* > > *Office: +1.303.962.7242* > > *Mobile: +1.720.319.8617* > > *Email: bcar...@broadhop.com* > > > > *From:* Randy Wilson [mailto:randyedwil...@gmail.com] > *Sent:* Tuesday, April 12, 2011 8:29 AM > *To:* haproxy@formilux.org > *Subject:* Tproxy with multiple interfaces > > > > Hi, > > I'm trying to setup an HAProxy instance to transparently load balance a > group of web servers. The HAProxy server and web servers each have two > interfaces; eth0 as the public interface and eth1 the private. I'm trying to > configure the load balancer to accept requests on port 80 on eth0 and > transparently proxy the connections to the web servers over the private > interfaces on eth1. I've configured the load balancer in the normal way for > tproxy, and have the web servers routing out through it. > > i.e. > > iptables -t mangle -N DIVERT > iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT > iptables -t mangle -A DIVERT -j MARK --set-mark 1 > iptables -t mangle -A DIVERT -j ACCEPT > > ip rule add fwmark 1 dev eth0 lookup 100 > ip rule add fwmark 1 dev eth1 lookup 100 > ip route add local 0.0.0.0/0 dev lo table 100 > > root@haproxy:~# sysctl net.ipv4.ip_forward > net.ipv4.ip_forward = 1 > > root@web:~# route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric RefUse > Iface > xxx.xxx.97.00.0.0.0 255.255.255.0 U 0 00 > eth0 > 192.168.0.0 0.0.0.0 255.255.252.0 U 0 00 > eth1 > 0.0.0.0 xxx.xxx.97.155 0.0.0.0 UG0 00 > eth0 > > ... > server web1 192.168.3.65:80 source 192.168.3.64 usesrc clientip > ... > > When testing this setup, the connection is correctly proxied to the web > server through eth1 and a response is sent back to the HAProxy server on > eth0, but it's ignored and the connection hangs. > > > Here's some netstat output during the connection: > > HAProxy: > > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program name > tcp0 1 xxx.xxx.48.1:42424 192.168.3.65:80 > SYN_SENT1386/haproxy > tcp0 0 xxx.xxx.97.155:80 xxx.xxx.48.1:42424 > ESTABLISHED 1386/haproxy > > Web Server: > > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program name > tcp
RE: Tproxy with multiple interfaces
Randy, The problem is the gateway on the backend webservers needs to be set as a VIP (or eth1 interface) on the HAproxy servers on their private interface (assuming you have two HAproxy servers and are using heartbeat for failover). It looks like from your routing table that eth0 on the webservers' gateway is pointed to the eth0 interface on haproxy this is why it works perfectly when you configure haproxy to use the public IPs on the webservers. Once you change the default gateway on the backend webserver's to use eth1 on the haproxy server (or a VIP which lives on eth1 using heartbeat for failover between two haproxy servers) then it will work. Brian Carpio Senior Systems Engineer [cid:image001.jpg@01CBF8F3.6C9226D0] Office: +1.303.962.7242 Mobile: +1.720.319.8617 Email: bcar...@broadhop.com From: Randy Wilson [mailto:randyedwil...@gmail.com] Sent: Tuesday, April 12, 2011 8:29 AM To: haproxy@formilux.org Subject: Tproxy with multiple interfaces Hi, I'm trying to setup an HAProxy instance to transparently load balance a group of web servers. The HAProxy server and web servers each have two interfaces; eth0 as the public interface and eth1 the private. I'm trying to configure the load balancer to accept requests on port 80 on eth0 and transparently proxy the connections to the web servers over the private interfaces on eth1. I've configured the load balancer in the normal way for tproxy, and have the web servers routing out through it. i.e. iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 dev eth0 lookup 100 ip rule add fwmark 1 dev eth1 lookup 100 ip route add local 0.0.0.0/0<http://0.0.0.0/0> dev lo table 100 root@haproxy:~# sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 root@web:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface xxx.xxx.97.00.0.0.0 255.255.255.0 U 0 00 eth0 192.168.0.0 0.0.0.0 255.255.252.0 U 0 00 eth1 0.0.0.0 xxx.xxx.97.155 0.0.0.0 UG0 00 eth0 ... server web1 192.168.3.65:80<http://192.168.3.65:80> source 192.168.3.64 usesrc clientip ... When testing this setup, the connection is correctly proxied to the web server through eth1 and a response is sent back to the HAProxy server on eth0, but it's ignored and the connection hangs. Here's some netstat output during the connection: HAProxy: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 1 xxx.xxx.48.1:42424 192.168.3.65:80<http://192.168.3.65:80> SYN_SENT1386/haproxy tcp0 0 xxx.xxx.97.155:80 xxx.xxx.48.1:42424 ESTABLISHED 1386/haproxy Web Server: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 192.168.3.65:80<http://192.168.3.65:80> xxx.xxx.48.1:42424 SYN_RECV- And the relevant entries from a tcpdump on each interface on each server: HAProxy/eth0 IP xxx.xxx.48.1.42424 > xxx.xxx.97.155.80: Flags [S], seq 1021535895, win 5840, options [mss 1418,sackOK,TS val 2448718 ecr 0,nop,wscale 7], length 0 HAProxy/eth0 IP xxx.xxx.97.155.80 > xxx.xxx.48.1.42424: Flags [S.], seq 504489330, ack 1158274356, win 5792, options [mss 1460,sackOK,TS val 230477 ecr 1407043,nop,wscale 7], length 0 HAProxy/eth0 IP xxx.xxx.48.1.42424 > xxx.xxx.97.155.80: Flags [.], ack 1, win 46, options [nop,nop,TS val 1407043 ecr 230477], length 0 HAproxy/eth1 IP xxx.xxx.48.1.42424 > 192.168.3.65.80: Flags [S], seq 391399045, win 5840, options [mss 1460,sackOK,TS val 230550 ecr 0,nop,wscale 7], length 0 Web/eth1 IP xxx.xxx.48.1.42424 > 192.168.3.65.80: Flags [S], seq 391399045, win 5840, options [mss 1460,sackOK,TS val 230550 ecr 0,nop,wscale 7], length 0 Web/eth0 IP 192.168.3.65.80 > xxx.xxx.48.1.42424: Flags [S.], seq 4033028970, ack 391399046, win 5792, options [mss 1460,sackOK,TS val 6751967 ecr 230550,nop,wscale 7], length 0 HAproxy/eth0 IP 192.168.3.65.80 > xxx.xxx.48.1.42424: Flags [S.], seq 4033028970, ack 391399046, win 5792, options [mss 1460,sackOK,TS val 6751967 ecr 230550,nop,wscale 7], length 0 NB. In this example I had set the usesrc setting to client so the client's port was used for readability, but the same occurs with clientip. I'm sure this is occurring because the response connection is arriving on a different interface to the one HAProxy originated the connection to the web server on. Does anyone know of a way around this? Is there an iptables or ip rule that can be set to switch the return traffic from eth0 to eth1? I have tried testing the setup by proxying the connections to the web server's pu
Tproxy with multiple interfaces
Hi, I'm trying to setup an HAProxy instance to transparently load balance a group of web servers. The HAProxy server and web servers each have two interfaces; eth0 as the public interface and eth1 the private. I'm trying to configure the load balancer to accept requests on port 80 on eth0 and transparently proxy the connections to the web servers over the private interfaces on eth1. I've configured the load balancer in the normal way for tproxy, and have the web servers routing out through it. i.e. iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 dev eth0 lookup 100 ip rule add fwmark 1 dev eth1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 root@haproxy:~# sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 root@web:~# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface xxx.xxx.97.00.0.0.0 255.255.255.0 U 0 00 eth0 192.168.0.0 0.0.0.0 255.255.252.0 U 0 00 eth1 0.0.0.0 xxx.xxx.97.155 0.0.0.0 UG0 00 eth0 ... server web1 192.168.3.65:80 source 192.168.3.64 usesrc clientip ... When testing this setup, the connection is correctly proxied to the web server through eth1 and a response is sent back to the HAProxy server on eth0, but it's ignored and the connection hangs. Here's some netstat output during the connection: HAProxy: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 1 xxx.xxx.48.1:42424 192.168.3.65:80 SYN_SENT1386/haproxy tcp0 0 xxx.xxx.97.155:80 xxx.xxx.48.1:42424 ESTABLISHED 1386/haproxy Web Server: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 192.168.3.65:80 xxx.xxx.48.1:42424 SYN_RECV- And the relevant entries from a tcpdump on each interface on each server: HAProxy/eth0 IP xxx.xxx.48.1.42424 > xxx.xxx.97.155.80: Flags [S], seq 1021535895, win 5840, options [mss 1418,sackOK,TS val 2448718 ecr 0,nop,wscale 7], length 0 HAProxy/eth0 IP xxx.xxx.97.155.80 > xxx.xxx.48.1.42424: Flags [S.], seq 504489330, ack 1158274356, win 5792, options [mss 1460,sackOK,TS val 230477 ecr 1407043,nop,wscale 7], length 0 HAProxy/eth0 IP xxx.xxx.48.1.42424 > xxx.xxx.97.155.80: Flags [.], ack 1, win 46, options [nop,nop,TS val 1407043 ecr 230477], length 0 HAproxy/eth1 IP xxx.xxx.48.1.42424 > 192.168.3.65.80: Flags [S], seq 391399045, win 5840, options [mss 1460,sackOK,TS val 230550 ecr 0,nop,wscale 7], length 0 Web/eth1 IP xxx.xxx.48.1.42424 > 192.168.3.65.80: Flags [S], seq 391399045, win 5840, options [mss 1460,sackOK,TS val 230550 ecr 0,nop,wscale 7], length 0 Web/eth0 IP 192.168.3.65.80 > xxx.xxx.48.1.42424: Flags [S.], seq 4033028970, ack 391399046, win 5792, options [mss 1460,sackOK,TS val 6751967 ecr 230550,nop,wscale 7], length 0 HAproxy/eth0 IP 192.168.3.65.80 > xxx.xxx.48.1.42424: Flags [S.], seq 4033028970, ack 391399046, win 5792, options [mss 1460,sackOK,TS val 6751967 ecr 230550,nop,wscale 7], length 0 NB. In this example I had set the usesrc setting to client so the client's port was used for readability, but the same occurs with clientip. I'm sure this is occurring because the response connection is arriving on a different interface to the one HAProxy originated the connection to the web server on. Does anyone know of a way around this? Is there an iptables or ip rule that can be set to switch the return traffic from eth0 to eth1? I have tried testing the setup by proxying the connections to the web server's public interface on eth0 instead: ... server web1 xxx.xxx.97.156:80 source xxx.xxx.97.155 usesrc clientip ... And the transparent proxying works perfectly. Any thoughts or suggestions appreciated. Many thanks, REW
