After changing the default gateway of the web servers to 10.10.130.79 this didnt fix it. The site we were testing on, and then all the other sites as well were unresponsive. So what I was unclear on is if we changed the default gateway to the vip of the test site we were using on the web server, how would the other web sites served from the box work. We have 4 sites on that box all w/ different VIPs for each. So we expected the other sites to fail and perhaps the test site to succeed but this wasnt the case. In the case of the test site traffic was getting to the web server to haproxy but not returning to either haproxy or the workstation making the request.
Id just like to clarify I few of my assumptions about this doc: http://blog.haproxy.com/2013/09/16/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer/ "Linux Kernel requirements You have to ensure your kernel has been compiled with the following options: – CONFIG_NETFILTER_TPROXY – CONFIG_NETFILTER_XT_TARGET_TPROXY" ^^^^ this to be done on haproxy boxes (not the webservers), ie: [richv@haproxy2 ~]$ lsmod | grep -i tproxy xt_TPROXY 17327 0 nf_defrag_ipv6 34651 2 xt_socket,xt_TPROXY nf_defrag_ipv4 12729 3 xt_socket,xt_TPROXY,nf_conntrack_ipv4 and: [richv@haproxy2 ~]$ grep -i tproxy /boot/* /boot/config-3.10.0-229.4.2.el7.x86_64:CONFIG_NETFILTER_XT_TARGET_TPROXY=m ** note, im using centos 7. in boot file i see CONFIG_NETFILTER_XT_TARGET_TPROXY in lsmod output only see xt_TPROXY. This is correct, I should see both CONFIG_NETFILTER_TPROXY & CONFIG_NETFILTER_XT_TARGET_TPROXY in lsmod output or boot file? ---------------------------------------------------------------------------------------------------------------- "sysctl settings The following sysctls must be enabled: – net.ipv4.ip_forward – net.ipv4.ip_nonlocal_bind" ^^^^ this to be done on haproxy boxes (not the webservers), ie: [richv@haproxy2 ~]$ sudo sysctl -p vm.swappiness = 0 net.ipv4.ip_nonlocal_bind = 1 net.ipv4.ip_forward = 1 --------------------------------------------------------------------------------------------------------------- "iptables rules You must setup the following iptables rules: iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT" ^^^^ this to be done on haproxy boxes (not the webservers), ie: haproxy2> sudo iptables -L -n -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination DIVERT tcp -- 0.0.0.0/0 0.0.0.0/0 socket [...] Chain DIVERT (1 references) target prot opt source destination MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set 0x1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ---------------------------------------------------------------------------------------------------------------- "IP route rules Then, tell the Operating System to forward packets marked by iptables to the loopback where HAProxy can catch them: ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100" ^^^^ this to be done on haproxy boxes (not the webservers), ie: haproxy2> ip rule show 0: from all lookup local 32762: from all fwmark 0x1 lookup 100 32766: from all lookup main 32767: from all lookup default haproxy> ip route show table 100 local default dev lo scope host ---------------------------------------------------------------------------------------------------------------- In summary for my setup, everything in that tutorial is to be performed on the haproxy box, not the web servers? ________________________________________ From: Baptiste <bed...@gmail.com> Sent: Friday, August 14, 2015 1:07 AM To: Rich Vigorito Cc: HAProxy Subject: Re: getting transparent proxy to work. "temporary" just for the troubleshooting period, and validate this is the root of your issue. The definitive solution belongs to you then! Please clarify the rest of your email. I don't understand what IPs or loopbacks you're speaking about. Before going further, please apply the default gateway change and confirm it works after this. Baptiste On Thu, Aug 13, 2015 at 10:28 PM, Rich Vigorito <ri...@ocp.org> wrote: > A couple clarifications. What do you mean by "temporary?" ... this wouldnt be > needed indefinitely? What ive articulated is only one site served through the > 2 web servers. Our web servers serve multiple sites, how to accommodate this? > Ie couldnt have 5 different IPs in the loopback? > ________________________________________ > From: Baptiste <bed...@gmail.com> > Sent: Wednesday, August 12, 2015 11:41 PM > To: Rich Vigorito > Cc: HAProxy > Subject: Re: getting transparent proxy to work. > > Hi Rich, > > so here is your problem. > Please temporarily change this default gateway of the web servers to > the active VIP: 10.10.130.79. > What happens, and what you highlithed in your diagrams is that HAProxy > creates the TCP connection with the client IP. > by default, the server tries to talk to the client directly, but the > client is not aware of HAProxy's connection and it refuses it. > If you route back your traffic to HAProxy, then HAProxy will handle > this connection and perform the relation with the real client. > > More information here: > http://blog.haproxy.com/2011/08/03/layer-7-load-balancing-transparent-proxy-mode/ > > Baptiste > > > On Thu, Aug 13, 2015 at 2:29 AM, Rich Vigorito <ri...@ocp.org> wrote: >> No inside the firewall one default gateway. 10.10.130.1 >> >> The web servers and haproxy servers have one interface I believe >> >> Sent from my Verizon Wireless 4G LTE DROID >> >> >> Baptiste <bed...@gmail.com> wrote: >> >> Do you mean your web servers have 2 interfaces, each one with its own >> default gateway? >> >> Baptiste >> >> Le 12 août 2015 23:10, "Rich Vigorito" <ri...@ocp.org> a écrit : >>> >>> Good to hear. Into the firewall 192.168.0.1 and out of the firewall >>> 10.10.130.1 >>> Thanks! >>> >>> Sent from my Verizon Wireless 4G LTE DROID >>> >>> >>> Baptiste <bed...@gmail.com> wrote: >>> >>> Hi Rich, >>> >>> Thanks a lot for this info, this is clearer now. >>> In my first mail, I asked you to provide us the default gateway of the >>> web servers. >>> could you please let us know this information ? >>> >>> Baptiste >>> >>> >>> On Wed, Aug 12, 2015 at 5:54 PM, Rich Vigorito <ri...@ocp.org> wrote: >>> > Also for clarification, the config listed in here is the config i used. >>> > The only difference between the 2 tests is removing: >>> > >>> > source 0.0.0.0 usesrc clientip >>> > >>> > Removing it loadbalancing works, keeping it in the config, load >>> > balancing doesnt work >>> > >>> > -Rich >>> > ________________________________________ >>> > From: Rich Vigorito <ri...@ocp.org> >>> > Sent: Monday, August 10, 2015 5:22 PM >>> > To: Baptiste >>> > Cc: haproxy@formilux.org >>> > Subject: RE: getting transparent proxy to work. >>> > >>> > Thanks you very much for all the help, and yes, you were correct about >>> > the capture i reported being the health check. attached are 2 pngs. one w/ >>> > our simple diagram of network topology and the other being what me and the >>> > network admin though was happening in our TCP handshake. This was >>> > determined >>> > by loading a tcpdump into wireshark. Those 2 files are dump.pcap (Which >>> > was >>> > on haproxy box) and web1_dump.pcap which was taking on the web server). >>> > What is happening is I dont think web server knows how to communicate to >>> > back to the haproxy box. the iptables rules and the ip rule and ip route >>> > commands from the blog post, in my set up would that be done on the >>> > haproxy >>> > boxes or the web servers? >>> > ________________________________________ >>> > From: Baptiste <bed...@gmail.com> >>> > Sent: Saturday, August 8, 2015 8:38 AM >>> > To: Rich Vigorito >>> > Cc: haproxy@formilux.org >>> > Subject: Re: getting transparent proxy to work. >>> > >>> > On Fri, Aug 7, 2015 at 11:05 PM, Rich Vigorito <ri...@ocp.org> wrote: >>> >> Hello, this is my first time using the mailing list. I have the >>> >> following >>> >> issue. >>> >> >>> >> >>> >> Followed steps to enable transparent proxy outlined here: >>> >> >>> >> Howto transparent proxying and binding with HAProxy and ALOHA >>> >> Load-Balancer >>> >> | HAProxy Technologies – Aloha Load Balancer >>> >> >>> >> >>> >> It will not load balance however w/ the following line added: >>> >> >>> >> >>> >> source 0.0.0.0 usesrc clientip >>> >> >>> >> Here is all the configuration and setup relevent: >>> >> >>> >> >>> >> bash> lsmod | grep -i tproxy >>> >> xt_TPROXY 17327 0 >>> >> nf_defrag_ipv6 34651 2 xt_socket,xt_TPROXY >>> >> nf_defrag_ipv4 12729 3 xt_socket,xt_TPROXY,nf_conntrack_ipv4 >>> >> >>> >> bash>sudo sysctl -p >>> >> vm.swappiness = 0 >>> >> net.ipv4.ip_nonlocal_bind = 1 >>> >> net.ipv4.ip_forward = 1 >>> >> >>> >> bash> sudo iptables -L -n -t mangle >>> >> Chain PREROUTING (policy ACCEPT) >>> >> target prot opt source destination >>> >> DIVERT tcp -- 0.0.0.0/0 0.0.0.0/0 socket >>> >> [...] >>> >> Chain DIVERT (1 references) >>> >> target prot opt source destination >>> >> MARK all -- 0.0.0.0/0 0.0.0.0/0 MARK set >>> >> 0x1 >>> >> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 >>> >> >>> >> bash> ip rule show >>> >> 0: from all lookup local >>> >> 32762: from all fwmark 0x1 lookup 100 >>> >> 32766: from all lookup main >>> >> 32767: from all lookup default >>> >> >>> >> bash> ip route show table 100 >>> >> local default dev lo scope host >>> >> >>> >> #haproxy.cfg >>> >> frontend layer4-listener >>> >> bind *:80 transparent >>> >> bind *:443 transparent >>> >> bind *:3306 >>> >> bind *:8080 >>> >> mode tcp >>> >> option tcplog >>> >> http-request set-header X-Forwarded-Proto https if { ssl_fc } >>> >> http-request set-header X-Forwarded-Proto http if !{ ssl_fc } >>> >> acl is_esp dst 10.10.130.79 >>> >> acl is_tls dst_port 443 >>> >> use_backend site_http if is_esp !is_tls >>> >> use_backend site_https if is_esp is_tls >>> >> backend site_https >>> >> mode tcp >>> >> option tcpka >>> >> option tcp-check >>> >> #source 0.0.0.0 usesrc clientip ## load balancing only works when >>> >> commented >>> >> out >>> >> server site_www1 www1.site.org:443 weight 1 check inter 2000 rise 2 >>> >> fall 3 >>> >> server site_www2 www2.site.org:443 weight 1 check inter 2000 rise 2 >>> >> fall 3 >>> >> >>> >> bash> haproxy -vv >>> >> HA-Proxy version 1.5.4 2014/09/02 >>> >> Copyright 2000-2014 Willy Tarreau <w...@1wt.eu> >>> >> Build options : >>> >> TARGET = linux2628 >>> >> CPU = generic >>> >> CC = gcc >>> >> CFLAGS = -O2 -g -fno-strict-aliasing >>> >> OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 >>> >> USE_PCRE=1 >>> >> >>> >> bash> uname -r >>> >> 3.10.0-229.4.2.el7.x86_64 >>> >> >>> >> >>> >> Our network admin was indicated the following: >>> >> >>> >> >>> >> A SYN packet from 10.10.130.31 (haproxy2) to 10.10.130.152 (site on >>> >> web1) >>> >> A SYN-ACK packet from web1 back to haproxy2 >>> >> A RST packet from haproxy2 to web1. >>> >> >>> >> >>> >> Anyone able/willing to help and/or give insight into this issue? >>> >> >>> >> >>> >> Thanks >>> > >>> > >>> > Hi Rich, >>> > >>> > the information you provide are quite inaccurate. >>> > I've already reported this on stackoverflow where you first posted >>> > your question. >>> > >>> > Here, for example, you ran multiple tests, with different >>> > configurations but you don't tell us during which one did your network >>> > admin saw the network he described. >>> > >>> > First point, the network packets reported by your network admin seems >>> > to be a health check... >>> > Second, it is hard to help troubleshooting transparent proxy without a >>> > network diagram. So please draw and share the simplest one showing a >>> > client, haproxy and a server, with their respective interfaces, IPs >>> > and default gateway. >>> > >>> > Last, a TCPdump on HAProxy box showing the traffic on the interface >>> > between haproxy and the server for the IP address of the client. >>> > >>> > Baptiste
