Hi All / Developers,Seams i have tranparent proxying working now on FreeBSD 8.3 with HAProxy1.5dev18 + small modification. Needed to add a firewall forwarding rule to forward the traffic to the localhost for socket processing.
Could a developer please make the following change? /*/* Add the following on line 33 of /include/common/compact.h */*//* *//*#ifdef *//*IP_BINDANY*//* *//* /* FreeBSD define variables */*//* *//* #define SOL_IP IPPROTO_IP*//* *//* #define SOL_IPV6 IPPROTO_IPV6*//* *//* #define IP_TRANSPARENT IP_BINDANY*//* *//*#endif*/After this haproxy can be successfully compiled on FreeBSD8.3 with the USE_LINUX_TPROXY=yes build option and transparent proxying works when the fwd firewall rule is made active.
On my pfSense2.1 system the following worked to load ipfw and add the fwd rule in ipfw.. :
/sbin/kldload ipfw/sbin/sysctl net.inet.ip.pfil.inbound="pf" net.inet6.ip6.pfil.inbound="pf" net.inet.ip.pfil.outbound="pf" net.inet6.ip6.pfil.outbound="pf"
/sbin/sysctl net.link.ether.ipfw=1 ipfw_context -a haproxy ipfw_context -s haproxy ipfw_context -a haproxy -n em0*ipfw -x haproxy add 20 fwd localhost tcp from IP-BACKEND-SERVER 80 to any in recv em0*
(This firewall rule should actually also check if the correct 'uid' of the haproxy process is set to also allow directly contacting the backendserver, but i could not get that part to work though that is not a HAProxy issue so should get fixed elsewhere.) And ideally it should be possible with 'pf' instead of 'ipfw', but that is still something i'm trying to investigate..
If this is not the correct way to fix/change this for FreeBSD could someone please advice on what is.?
Thanks in advance. PiBa-NL Op 18-4-2013 23:46, PiBa-NL schreef:
Hi All / Baptiste,It seams i have found error in my initial email sorry for that. It should have mentioned IP_BINDANY in the example line of code instead of IP_FREEBIND. Still the my problem remains the same.Returning SYN-ACK packets are not recieved/intercepted by HAProxy. And the browser gets a "No server is available to handle this request." after exactly 2 minutes.There seam to be needed at least two changes for compiling/using the full-transparent-proxy feature off FreeBSD.In the file *compat.h* (or anywhere that fits better.) add: /*#ifndef SOL_IP** **#define SOL_IP 0** **#define SOL_IPV6 0** **#endif*/* * In the file *proto_tcp.c* change: /if (flags && ip_transp_working) {// // if (// ///*#ifdef *////*/*IP_BINDANY*/** */ //*setsockopt(fd, SOL_IP, IP_BINDANY, &one, sizeof(one))*//*|| *// ////*#endif*/* * /setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0//// || setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one)) == 0)//// {// //// // foreign_ok = 1;// // }// // else// // ip_transp_working = 0;// // }/This makes it work for the sending a SYN connection request to the backend server... However the SYN-ACK response, though it is passing through the HAProxy server, is not recieved by HAProxy.A tcpdump with mac-addresses on the LAN side of the FreeBSD router shows shows that .1.50 contacts haproxy on .1.22:81, gets a SYN-ACK reply back, but after that also SYN-ACK packets from the backend server.. as can be seen from the MAC adresses all recieved traffic comes through the FreeBSD machine: 21:03:06.018602 00:0c:29:0d:89:90 > 00:0c:29:b5:f0:fe, ethertype IPv4 (0x0800), length 62: 192.168.1.50.1177 > 192.168.1.22.81: Flags [S], seq 252352307, win 64240, options [mss 1460,nop,nop,sackOK], length 0 21:03:06.018735 00:0c:29:b5:f0:fe > 00:0c:29:0d:89:90, ethertype IPv4 (0x0800), length 62: 192.168.1.22.81 > 192.168.1.50.1177: Flags [S.], seq 2295999072, ack 252352308, win 65228, options [mss 1460,sackOK,eol], length 0 21:03:06.019066 00:0c:29:0d:89:90 > 00:0c:29:b5:f0:fe, ethertype IPv4 (0x0800), length 60: 192.168.1.50.1177 > 192.168.1.22.81: Flags [.], ack 1, win 64240, length 0 21:03:06.020429 00:0c:29:0d:89:90 > 00:0c:29:b5:f0:fe, ethertype IPv4 (0x0800), length 336: 192.168.1.50.1177 > 192.168.1.22.81: Flags [P.], ack 1, win 64240, length 282 21:03:06.020517 00:0c:29:b5:f0:fe > 00:0c:29:0d:89:90, ethertype IPv4 (0x0800), length 54: 192.168.1.22.81 > 192.168.1.50.1177: Flags [.], ack 283, win 65418, length 0 21:03:06.021205 00:0c:29:b5:f0:fe > 00:0c:29:0d:89:90, ethertype IPv4 (0x0800), length 74: 192.168.0.40.81 > 192.168.1.50.49496: Flags [S.], seq 2271887215, ack 4020735996, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 144738 ecr 92963], length 0 21:03:09.026308 00:0c:29:b5:f0:fe > 00:0c:29:0d:89:90, ethertype IPv4 (0x0800), length 74: 192.168.0.40.81 > 192.168.1.50.49496: Flags [S.], seq 2271887215, ack 4020735996, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 145038 ecr 92963], length 0 21:03:15.025375 00:0c:29:b5:f0:fe > 00:0c:29:0d:89:90, ethertype IPv4 (0x0800), length 70: 192.168.0.40.81 > 192.168.1.50.49496: Flags [S.], seq 2271887215, ack 4020735996, win 8192, options [mss 1260,sackOK,TS val 145638 ecr 92963], length 0Those last three packets should not go towards the browser pc.. Op 17-4-2013 21:55, PiBa-NL schreef:Hi Baptiste,Thanks for your reply, i understand that the traffic must pass through the router/HAProxy box, and for that part i can confirm that the routing of packets is 'working' like it should.To explain my setup a little more:*Webserver* (Win7) 192.168.0.40/24 on a OPT1/DMZ using gateway :192.168.0.117*HAProxy* on FreeBSD 8.3(pfSense2.1) which performs routing between the 2 networks has 2 interfaces(that matter):DMZ interface has 192.168.0.117/24 LAN interface has 192.168.1.1/24 *ClientPC* (WinXP) 192.168.1.50/24 on the LAN gateway :192.168.1.1(p.s. to make the picture complete and a little more complicated the FreeBSD and ClientPC are running within 'VMware Workstation' which runs on the Win7 machine., But im positive the networks are correctly separated (using a 'LAN-segment' between FreeBSD and the ClientPC) and traffic does flow through the FreeBSD machine.) All traffic moving between clientPC and the Webserver 'must' go through the FreeBSD machine which routes the traffic, i have confirmed that traffic moves correctly through the FreeBSD machine by using wireshark and tcpdump on all machines.So i have traced the packets of the connection to go like this: 1-ClientPC browser 'connects' to HAProxy and waits for the webpage (ok) 2-HAProxy sends SYN packet to Webserver with a spoofed source IP (ok)3-Webserver sends response SYN-ack back to the spoofed source IP to its gateway 192.168.0.117 (and the MAC address using that IP) (expected) 4-FreeBSD passes the SYN-ack on to the ClientPC.(this is what should not happen.) ## HAproxy waits and retries for a few timeouts to occur.. and then eventually after about 2 minutes the browser is served the 503 page "No server is available to handle this request." ##So to me it seams HAproxy should in step 4 somehow get the SYN-ack packet, im not sure if it should listen/bind on all local IP's or if the traffic should get passed back to a HAProxy port by using for example a NAT rule.?.PiBa-NL Op 17-4-2013 21:21, Baptiste schreef:Hi, In order to work in transparent proxy mode, the server's default gateway must be the HAproxy server. Or at least, the traffic from the server must pass through the haproxy box before reaching the client. Even if HAProxy spoof the client IP, it's HAProxy which initializes the TCP connection, so if the server try to reach the client directly, this one would refuse the connection. Baptiste On Wed, Apr 17, 2013 at 8:01 PM, PiBa-NL<[email protected]> wrote:I forgot to mention im using HAproxy 1.5dev18. Hello HAProxy developers/users, I would like to be able to run HAProxy transparently on FreeBSD 8.3. This would be both for my own usage and also to make it available to a larger public by including it in a 'haproxy-devel' package for pfSense. However when trying to use it i get the error: [ALERT] 104/235847 (72477) : parsing [/var/etc/haproxy.cfg:34] : 'usesrc' not allowed here because support for TPROXY was not compiled in. From what i read it seams it should be possible. For example the Makefile contains the following: ifeq ($(TARGET),freebsd) USE_TPROXY = implicit Which seams like it is supposed to be 'supported'. Ive also tried the USE_LINUX_TPROXY=yes compile flag, but this returns 2 undeclared variables SOL_IP and SOL_IPV6. Ive tried declaring them with substitute values like 'IP_BINDANY', or the value 6 which could stand for the TCP protocol type, or 0. , but though the source did then compile the end result still was that either an error was returned to the browser that no backend was available, together with the following debug error: [ALERT] 104/235129 (17380) : Cannot bind to tproxy source address before connect() for backend pb3TEST_http. Aborting. Or i dont get a response at all and HAproxy seems to be waiting for 'something' to happen.. Could it be that something is not fully supported in HAProxy toghether with FreeBSD to allow transparent proxying? Or am i looking at the wrong side of the problem and would i need to compile the FreeBSD kernel with tproxy support.? Which I believe would be natively supported in version 8, but i might be wrong on that.. I i add after "setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)" this line: setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one)); It removes the error about 'Cannot bind to tproxy source address...' and packets do seam to be send to the proper destination. Except the connection never establishes.. The browser running on 192.168.1.50 contacts haproxy on its IP:port http://192.168.1.22:81/ Haproxy then forwards the traffic to the server 192.168.0.40.81 which is according to status page "L7OK/200 in 0ms". Also the reply packets gets routed back to the original client pc (wireshark confirmed that..), and seam not to get intercepted by HAproxy which i think is supposed to happen.?.. when passing through the 'FreeBSD router'. But when performing a tcpdump on the interface in the 192.168.0.117 network only SYN and SYN-ack packets seem to be 'exchanged'.. Bet never any actual 'data' 21:02:04.915310 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:04.915464 IP 192.168.0.40.81 > 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281557 ecr 1556876], length 0 21:02:04.915546 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:07.910690 IP 192.168.0.40.81 > 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281857 ecr 1556876], length 0 21:02:07.911073 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:07.911079 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:11.110673 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:11.110685 IP 192.168.1.50.51194 > 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:13.913959 IP 192.168.0.40.81 > 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,sackOK,TS val 1282457 ecr 1556876], length 0 While when the pc contacts the webserver directly it works 'normally'.. 20:45:12.746359 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [S], seq 24635592, win 64240, options [mss 1460,nop,nop,sackOK], length 0 20:45:12.746473 IP 192.168.0.40.81 > 192.168.1.50.3588: Flags [S.], seq 3353931105, ack 24635593, win 8192, options [mss 1260,nop,nop,sackOK], length 0 20:45:12.746937 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [.], ack 1, win 64260, length 0 20:45:12.747071 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [P.], ack 1, win 64260, length 282 20:45:12.750878 IP 192.168.0.40.81 > 192.168.1.50.3588: Flags [.], ack 283, win 65520, length 1260 20:45:12.751005 IP 192.168.0.40.81 > 192.168.1.50.3588: Flags [P.], ack 283, win 65520, length 1142 20:45:12.751463 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [.], ack 2403, win 64260, length 0 20:45:12.800179 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [P.], ack 2403, win 64260, length 297 20:45:12.800753 IP 192.168.0.40.81 > 192.168.1.50.3588: Flags [.], ack 580, win 65223, length 1260 20:45:12.800871 IP 192.168.0.40.81 > 192.168.1.50.3588: Flags [P.], ack 580, win 65223, length 151 20:45:12.801488 IP 192.168.1.50.3588 > 192.168.0.40.81: Flags [.], ack 3814, win 64260, length 0 See below my configuration of HAproxy: global maxconn 300 log /var/run/log local6 debug stats socket /tmp/haproxy.socket level admin nbproc 1 chroot /var/empty daemon frontend test_pb3 bind 192.168.1.22:81 mode http log global option dontlognull maxconn 444 timeout client 30000 default_backend pb3TEST_http backend pb3TEST_http mode http timeout connect 30000 timeout server 30000 retries 3 option httpchk OPTIONS / source 192.168.0.117 usesrc clientip server pb3_srv 192.168.0.40:81 check inter 10000 weight 1 Could someone give me advice on what might need to change, what to test or how i could proceed further with making it work ? Thanks in advance, PiBa-NL
