Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy,
Thanks the patches look good, and when applied separately all compile
without issues on FreeBSD. (Except when using the USE_LINUX_TPROXY flag,
but that shouldn't be used on FreeBSD anyway.)
And transparent proxying works correctly on FreeBSD as was expected.
Ive included a transparent_proxy.cfg which could be added to the
'examples' folder of HAProxy. Though i'm not sure anyone would find it
there.
Also it includes quite a bit of 'supposedly' better configuration hints,
that i'm currently unable to verify, but still might help someone in the
future..
As for Linux i think some iptable rules are needed. Maybe someone can
added those to the example.
If you don't want to include it in its current form or want to reformat
the whole thing i have no problem with that.
Thanks,
Pieter Baauw
Op 8-5-2013 23:54, Willy Tarreau schreef:
OK here's what I came up with. There are 3 patches :
- 0001 : reorganize flags processing
- 0002 : add support for freebsd
- 0003 : add support for openbsd
Please review and test if you can. At least it seems OK on linux here.
I have written all the commit messages. Feel free to change them if you
want, as they're made under your name. If you want to provide additional
doc, let's just add a 4th patch on top of this.
The code is not quite beautiful, but that's always the price to pay
when playing with ifdefs, and there are already a large number of them
in the same functions anyway.
Also, if you could provide a real name for the commits, it would be nice!
Thanks!
Willy
#
# This is an example of how to configure HAProxy to be used as a 'full
transparent proxy' for a single backend server.
#
# Note that to actually make this work extra firewall/nat rules are required.
# Also HAProxy needs to be compiled with support for this, in HAProxy1.5-dev19
you can check if this is the case with haproxy -vv.
#
global
frontend MyFrontend
bind192.168.1.22:80
default_backend TransparentBack_http
backend TransparentBack_http
modehttp
source 0.0.0.0 usesrc client
server MyWebServer 192.168.0.40:80
#
# To create the the nat rules perform the following:
#
# ### (FreeBSD 8) ###
# --- Step 1 ---
# ipfw is needed to get 'reply traffic' back to the HAProxy process, this can
be achieved by configuring a rule like this:
# fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# The following would be even better but this did not seam to work on the
pfSense2.1 distribution of FreeBSD 8.3:
# fwd 127.0.0.1:80 tcp from any 80 to any in recv ${outside_iface} uid
${proxy_uid}
#
# If only 'pf' is currently used some aditional steps are needed to load and
configure ipfw:
# You need to configure this to always run on startup:
#
# /sbin/kldload ipfw
# /sbin/sysctl net.inet.ip.pfil.inbound=pf net.inet6.ip6.pfil.inbound=pf
net.inet.ip.pfil.outbound=pf net.inet6.ip6.pfil.outbound=pf
# /sbin/sysctl net.link.ether.ipfw=1
# ipfw add 10 fwd localhost tcp from 192.168.0.40 80 to any in recv em0
#
# the above does the folowing:
# - load the ipfw kernal module
# - set pf as the outer firewall to keep control of routing packets for example
to route them to a non-default gateway
# - enable ipfw
# - set a rule to catches reply traffic on em0 comming from the webserver
#
# --- Step 2 ---
# To also make the client connection transparent its possible to redirect
incomming requests to HAProxy with a pf rule:
# rdr on em1 proto tcp from any to 192.168.0.40 port 80 - 192.168.1.22
# here em1 is the interface that faces the clients, and traffic that is
originally send straight to the webserver is redirected to HAProxy
#
# ### (FreeBSD 9) (OpenBSD 4.4) ###
# pf supports divert-reply which is probably better suited for the job
above then ipfw..
#
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy,
Could you please let me know what your findings are about the proposed
patch?
Does it need some more work, is it implemented wrongly, or would it help
if i send my current haproxy.cfg file?
If i need to change something please let me know, thanks.
Thanks for your time,
PiBa-NL
Op 3-5-2013 18:03, Willy Tarreau schreef:
Hi,
sorry, I missed it.
I'll give it a look and merge it if it's OK.
Thanks, Willy
Op 27-4-2013 18:08, PiBa-NL schreef:
Hi Willy,
I generated 2 patch files:
-FreeBSD IP_BINDANY git diff.patch generated with a git diff
(against a hopefully relatively recent source tree)(i couldnt get it
to fetch http://git.1wt.eu/git/haproxy.git ..)
-FreeBSD IP_BINDANY diff -urN.patch generated with diff -urN
(against the 'port source')
I hope one of them can be used by you.
Please take a look and comment if something is amiss.
Greetings
PiBa-NL
diff -urN workoriginal/haproxy-1.5-dev18/include/common/compat.h
work/haproxy-1.5-dev18/include/common/compat.h
--- workoriginal/haproxy-1.5-dev18/include/common/compat.h 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/include/common/compat.h 2013-04-27
14:56:27.0 +
@@ -93,6 +93,15 @@
#endif /* !IPV6_TRANSPARENT */
#endif /* CONFIG_HAP_LINUX_TPROXY */
+#if (defined(SOL_IP)defined(IP_TRANSPARENT)) \
+ || (defined(SOL_IPV6) defined(IPV6_TRANSPARENT)) \
+ || (defined(SOL_IP)defined(IP_FREEBIND)) \
+ || (defined(IPPROTO_IP)defined(IP_BINDANY)) \
+ || (defined(IPPROTO_IPV6) defined(IPV6_BINDANY)) \
+ || (defined(SOL_SOCKET)defined(SO_BINDANY))
+ #define HAP_TRANSPARENT
+#endif
+
/* We'll try to enable SO_REUSEPORT on Linux 2.4 and 2.6 if not defined.
* There are two families of values depending on the architecture. Those
* are at least valid on Linux 2.4 and 2.6, reason why we'll rely on the
diff -urN workoriginal/haproxy-1.5-dev18/include/types/connection.h
work/haproxy-1.5-dev18/include/types/connection.h
--- workoriginal/haproxy-1.5-dev18/include/types/connection.h 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/include/types/connection.h 2013-04-27
14:56:30.0 +
@@ -219,7 +219,7 @@
char *iface_name;/* bind interface name or NULL */
struct port_range *sport_range; /* optional per-server TCP source
ports */
struct sockaddr_storage source_addr; /* the address to which we want to
bind for connect() */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
struct sockaddr_storage tproxy_addr; /* non-local address we want to
bind to for connect() */
char *bind_hdr_name; /* bind to this header name if
defined */
int bind_hdr_len;/* length of the name of the
header above */
diff -urN workoriginal/haproxy-1.5-dev18/src/backend.c
work/haproxy-1.5-dev18/src/backend.c
--- workoriginal/haproxy-1.5-dev18/src/backend.c2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/src/backend.c2013-04-27 14:56:32.0
+
@@ -884,7 +884,7 @@
*/
static void assign_tproxy_address(struct session *s)
{
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
struct server *srv = objt_server(s-target);
struct conn_src *src;
diff -urN workoriginal/haproxy-1.5-dev18/src/cfgparse.c
work/haproxy-1.5-dev18/src/cfgparse.c
--- workoriginal/haproxy-1.5-dev18/src/cfgparse.c 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/src/cfgparse.c 2013-04-27 14:56:33.0
+
@@ -4535,8 +4535,8 @@
cur_arg += 2;
while (*(args[cur_arg])) {
if (!strcmp(args[cur_arg], usesrc)) {
/* address to use outside */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
-#if !defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
+#if !defined(HAP_TRANSPARENT)
if
(!is_addr(newsrv-conn_src.source_addr)) {
Alert(parsing [%s:%d]
: '%s' requires an explicit '%s' address.\n,
file, linenum,
usesrc, source);
@@ -4625,7 +4625,7 @@
newsrv-conn_src.opts
|= CO_SRC_TPROXY_ADDR;
}
global.last_checks |=
LSTCHK_NETADM;
-#if !defined(CONFIG_HAP_LINUX_TPROXY)
+#if !defined(HAP_TRANSPARENT)
global.last_checks |=
LSTCHK_CTTPROXY;
#endif
cur_arg += 2;
@@ -4635,7 +4635,7 @@
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Wed, May 08, 2013 at 07:34:19PM +0200, PiBa-NL wrote: Hi Willy, Could you please let me know what your findings are about the proposed patch? I was on it this afternoon (didn't have time earlier) :-) I haven't finished reviewing it yet, because I was trying to figure if there would be an easy way to merge the CTTPROXY mode into the other transparent proxy options, but I'm not sure that's really useful. Also I found one issue here : + int ret = 0; + #if defined(SOL_IP)defined(IP_TRANSPARENT) + ret |= setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0; + #endif + #if defined(SOL_IP)defined(IP_FREEBIND) + ret |= setsockopt(fd, SOL_IP, IP_FREEBIND, one, sizeof(one)) == 0; + #endif + #if defined(IPPROTO_IP)defined(IP_BINDANY) + ret |= setsockopt(fd, IPPROTO_IP, IP_BINDANY, one, sizeof(one)) == 0; + #endif + #if defined(SOL_SOCKET)defined(SO_BINDANY) + ret |= setsockopt(fd, SOL_SOCKET, SO_BINDANY, one, sizeof(one)) == 0; + #endif + if (ret) As you can see, if we have multiple defines, we'll call setsockopt multiple times, which we don't want. I was thinking about something like this instead : if (0 #if cond1 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0 #endif #if cond2 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0 #endif ) ... I'm still on it right now, to ensure we don't break anything. Does it need some more work, is it implemented wrongly, or would it help if i send my current haproxy.cfg file? If i need to change something please let me know, thanks. I do not think so, I can easily perform the changes above myself, I won't harrass you with another iteration. Overall it's good but since we're changing many things at once, I'm cautious. I'd prefer to break it in two BTW : 1) change existing code to support CONFIG_HAP_TRANSPARENT everywhere 2) add FreeBSD support But if that's OK for you, I'll simply perform the small adjustments before merging it. Cheers, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy, If you make some changes to what you think/know is better and break the change into two parts is fine for me. About calling setsockopt multiple times, i think the ret |= would not evaluate the call behind it if ret already is 1, not absolutely sure about that.. I didn't think of starting a if statement with 0 || which might speed it up a clock tick or two so would be better anyway instead of having a variable assignment in between. Thanks, could you let me know when its ready then ill give it another compilecheck on FreeBSD. And provide a little 'documentation' on how i configured the 'ipfw' firewall/nat to make it work. p.s. Ive spotted a issue in my patch with the IPv6 part where i forgot about the OpenBSD part (SOL_SOCKET SO_BINDANY) should probably be added there also. PiBa-NL Op 8-5-2013 20:18, Willy Tarreau schreef: Hi, On Wed, May 08, 2013 at 07:34:19PM +0200, PiBa-NL wrote: Hi Willy, Could you please let me know what your findings are about the proposed patch? I was on it this afternoon (didn't have time earlier) :-) I haven't finished reviewing it yet, because I was trying to figure if there would be an easy way to merge the CTTPROXY mode into the other transparent proxy options, but I'm not sure that's really useful. Also I found one issue here : + int ret = 0; + #if defined(SOL_IP)defined(IP_TRANSPARENT) + ret |= setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0; + #endif + #if defined(SOL_IP)defined(IP_FREEBIND) + ret |= setsockopt(fd, SOL_IP, IP_FREEBIND, one, sizeof(one)) == 0; + #endif + #if defined(IPPROTO_IP)defined(IP_BINDANY) + ret |= setsockopt(fd, IPPROTO_IP, IP_BINDANY, one, sizeof(one)) == 0; + #endif + #if defined(SOL_SOCKET)defined(SO_BINDANY) + ret |= setsockopt(fd, SOL_SOCKET, SO_BINDANY, one, sizeof(one)) == 0; + #endif + if (ret) As you can see, if we have multiple defines, we'll call setsockopt multiple times, which we don't want. I was thinking about something like this instead : if (0 #if cond1 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0 #endif #if cond2 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0 #endif ) ... I'm still on it right now, to ensure we don't break anything. Does it need some more work, is it implemented wrongly, or would it help if i send my current haproxy.cfg file? If i need to change something please let me know, thanks. I do not think so, I can easily perform the changes above myself, I won't harrass you with another iteration. Overall it's good but since we're changing many things at once, I'm cautious. I'd prefer to break it in two BTW : 1) change existing code to support CONFIG_HAP_TRANSPARENT everywhere 2) add FreeBSD support But if that's OK for you, I'll simply perform the small adjustments before merging it. Cheers, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Wed, May 08, 2013 at 09:41:33PM +0200, PiBa-NL wrote: Hi Willy, If you make some changes to what you think/know is better and break the change into two parts is fine for me. OK. About calling setsockopt multiple times, i think the ret |= would not evaluate the call behind it if ret already is 1, not absolutely sure about that.. No, I can guarantee you that all of them will be called. I didn't think of starting a if statement with 0 || which might speed it up a clock tick or two so would be better anyway instead of having a variable assignment in between. It's not a matter of saving a clock cycle but really not to call the setsockopt we don't want to call, while still keeping the ability to fall back to the remaining supported ones when possible. Thanks, could you let me know when its ready then ill give it another compilecheck on FreeBSD. And provide a little 'documentation' on how i configured the 'ipfw' firewall/nat to make it work. Perfect, I'll send you the patch back before merging it then. p.s. Ive spotted a issue in my patch with the IPv6 part where i forgot about the OpenBSD part (SOL_SOCKET SO_BINDANY) should probably be added there also. No problem, we'll add this as a third patch. It's really important to have one feature per commit, because when for some reason we introduce regressions, users can easily revert just the faulty commit without losing the other ones. Cheers, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
p.s. Ive spotted a issue in my patch with the IPv6 part where i forgot about the OpenBSD part (SOL_SOCKET SO_BINDANY) should probably be added there also. No problem, we'll add this as a third patch. It's really important to have one feature per commit, because when for some reason we introduce regressions, users can easily revert just the faulty commit without losing the other ones. BTW, I noticed that we can safely remove the following tests : - defined(SOL_IP) - defined(SOL_IPV6) = both are used only on linux and were previously present - defined(IPPROTO_IP) - defined(IPPROTO_IPV6) = we already have many other IPPROTO_* defines (mainly IPPROTO_TCP) so we already have the correct includes to use them. - defined(SOL_SOCKET) = is already referenced a lot, so it's safe. It's nice because it will clear the #ifdef a little bit. Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
OK here's what I came up with. There are 3 patches :
- 0001 : reorganize flags processing
- 0002 : add support for freebsd
- 0003 : add support for openbsd
Please review and test if you can. At least it seems OK on linux here.
I have written all the commit messages. Feel free to change them if you
want, as they're made under your name. If you want to provide additional
doc, let's just add a 4th patch on top of this.
The code is not quite beautiful, but that's always the price to pay
when playing with ifdefs, and there are already a large number of them
in the same functions anyway.
Also, if you could provide a real name for the commits, it would be nice!
Thanks!
Willy
From fa20b9333c0ee547ec1214da73bdc2d7163442dd Mon Sep 17 00:00:00 2001
From: PiBa-NL piba.nl@gmail.com
Date: Wed, 8 May 2013 22:49:23 +0200
Subject: REORG: tproxy: prepare the transparent proxy defines for accepting
other OSes
This patch does not change the logic of the code, it only changes the
way OS-specific defines are tested.
At the moment the transparent proxy code heavily depends on Linux-specific
defines. This first patch introduces a new define CONFIG_HAP_TRANSPARENT
which is set every time the defines used by transparent proxy are present.
This also means that with an up-to-date libc, it should not be necessary
anymore to force CONFIG_HAP_LINUX_TPROXY during the build, as the flags
will automatically be detected.
The CTTPROXY flags still remain separate because this older API doesn't
work the same way.
A new line has been added in the version output for haproxy -vv to indicate
what transparent proxy support is available.
---
include/common/compat.h| 6 ++
include/types/connection.h | 2 +-
src/backend.c | 2 +-
src/cfgparse.c | 20 +--
src/haproxy.c | 16 +++
src/proto_tcp.c| 49 ++
6 files changed, 70 insertions(+), 25 deletions(-)
diff --git a/include/common/compat.h b/include/common/compat.h
index bb2d010..043a56e 100644
--- a/include/common/compat.h
+++ b/include/common/compat.h
@@ -93,6 +93,12 @@
#endif /* !IPV6_TRANSPARENT */
#endif /* CONFIG_HAP_LINUX_TPROXY */
+#if defined(IP_FREEBIND) \
+ || defined(IP_TRANSPARENT)\
+ || defined(IPV6_TRANSPARENT)
+#define CONFIG_HAP_TRANSPARENT
+#endif
+
/* We'll try to enable SO_REUSEPORT on Linux 2.4 and 2.6 if not defined.
* There are two families of values depending on the architecture. Those
* are at least valid on Linux 2.4 and 2.6, reason why we'll rely on the
diff --git a/include/types/connection.h b/include/types/connection.h
index 255811c..2c7acd1 100644
--- a/include/types/connection.h
+++ b/include/types/connection.h
@@ -219,7 +219,7 @@ struct conn_src {
char *iface_name;/* bind interface name or NULL */
struct port_range *sport_range; /* optional per-server TCP source
ports */
struct sockaddr_storage source_addr; /* the address to which we want to
bind for connect() */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
struct sockaddr_storage tproxy_addr; /* non-local address we want to
bind to for connect() */
char *bind_hdr_name; /* bind to this header name if
defined */
int bind_hdr_len;/* length of the name of the
header above */
diff --git a/src/backend.c b/src/backend.c
index 9f4e635..9f23018 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -884,7 +884,7 @@ int assign_server_and_queue(struct session *s)
*/
static void assign_tproxy_address(struct session *s)
{
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
struct server *srv = objt_server(s-target);
struct conn_src *src;
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 6f2850c..9907bfd 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -1813,7 +1813,7 @@ int cfg_parse_listen(const char *file, int linenum, char
**args, int kwm)
curproxy-conn_src.iface_name =
strdup(defproxy.conn_src.iface_name);
curproxy-conn_src.iface_len =
defproxy.conn_src.iface_len;
curproxy-conn_src.opts = defproxy.conn_src.opts;
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
curproxy-conn_src.tproxy_addr =
defproxy.conn_src.tproxy_addr;
#endif
}
@@ -4558,8 +4558,8 @@ stats_error_parsing:
cur_arg += 2;
while (*(args[cur_arg])) {
if (!strcmp(args[cur_arg], usesrc)) {
/* address to use outside */
-#if defined(CONFIG_HAP_CTTPROXY) ||
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy, It seams the changes where easier than i expected, assuming ive done it 'correctly'... I generated 2 patch files: -FreeBSD IP_BINDANY git diff.patch generated with a git diff (against a hopefully relatively recent source tree)(i couldnt get it to fetch http://git.1wt.eu/git/haproxy.git ..) -FreeBSD IP_BINDANY diff -urN.patch generated with diff -urN (against the 'port source') I hope one of them can be used by you. Please take a look and comment if something is amiss. Greetings PiBa-NL Op 26-4-2013 23:40, PiBa-NL schreef: Hi Willy, Ill give it a try and send the patch as an attachment, though im not 100% comfortable with the code. I think i can do it. Will take me a few days though.. Thanks sofar. Op 26-4-2013 23:12, Willy Tarreau schreef: On Fri, Apr 26, 2013 at 11:03:00PM +0200, PiBa-NL wrote: Hi Willy / Lukas, It seams to me OpenBSD doesn't support the IP_BINDANY flag..: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/netinet/in.h http://www.openbsd.org/cgi-bin/cvsweb/%7Echeckout%7E/src/sys/netinet/in.h it seems it has, but differently : http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-07/msg00399.html While FreeBSD does: http://svnweb.freebsd.org/base/head/sys/netinet/in.h?view=markup But then again neither of them supports SOL_IP, so i would expect compilation to simply 'fail'. When trying to compile with the USE_LINUX_TPROXY option. Which is exactly the reason I don't want to remap these things which are linux-specific, and instead use the proper call depending on the available flags. Eg something like this : #if defined(SOL_IP) defined(IP_TRANSPARENT) /* linux */ ret = setsockop(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* freebsd */ ret = setsockop(fd, IP_PROTOIP, IP_BINDANY, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* openbsd */ ret = setsockop(fd, SOL_SOCKET, SO_BINDANY, one, sizeof(one)); #else /* unsupported platform */ ret = -1; #endif The combination i think is unlikely to cause problems for other currently working builds/systems.. If you want i can probably come up with a combination that makes it work for FreeBSD with a special USE_FREEBSD_TPROXY make option. No, really I think something like above is much better for the long term. It's more work to adapt existing code first but will pay in the long term, even in the short term if it allows us to support OpenBSD at the same time. Or go for the 'full automatic inclusion' depending on available flags. Which i think is even 'nicer'. But probably needs more testing to confirm proper working.. I would be willing to make these changes. Is this the way to go? As you like, if you feel comfortable with changing the way the current code works (the linux-specific one), feel free to try, otherwise I can do it over the week-end, and then a second patch derived from yours will bring in support for FreeBSD then OpenBSD if someone here is able to test it. Thanks for reviewing my proposed changes sofar. you're welcome :-) Willy diff -urN workoriginal/haproxy-1.5-dev18/include/common/compat.h work/haproxy-1.5-dev18/include/common/compat.h --- workoriginal/haproxy-1.5-dev18/include/common/compat.h 2013-04-26 19:36:15.0 + +++ work/haproxy-1.5-dev18/include/common/compat.h 2013-04-27 14:56:27.0 + @@ -93,6 +93,15 @@ #endif /* !IPV6_TRANSPARENT */ #endif /* CONFIG_HAP_LINUX_TPROXY */ +#if (defined(SOL_IP)defined(IP_TRANSPARENT)) \ + || (defined(SOL_IPV6) defined(IPV6_TRANSPARENT)) \ + || (defined(SOL_IP)defined(IP_FREEBIND)) \ + || (defined(IPPROTO_IP)defined(IP_BINDANY)) \ + || (defined(IPPROTO_IPV6) defined(IPV6_BINDANY)) \ + || (defined(SOL_SOCKET)defined(SO_BINDANY)) + #define HAP_TRANSPARENT +#endif + /* We'll try to enable SO_REUSEPORT on Linux 2.4 and 2.6 if not defined. * There are two families of values depending on the architecture. Those * are at least valid on Linux 2.4 and 2.6, reason why we'll rely on the diff -urN workoriginal/haproxy-1.5-dev18/include/types/connection.h work/haproxy-1.5-dev18/include/types/connection.h --- workoriginal/haproxy-1.5-dev18/include/types/connection.h 2013-04-26 19:36:15.0 + +++ work/haproxy-1.5-dev18/include/types/connection.h 2013-04-27 14:56:30.0 + @@ -219,7 +219,7 @@ char *iface_name;/* bind interface name or NULL */ struct port_range *sport_range; /* optional per-server TCP source ports */ struct sockaddr_storage source_addr; /* the address to which we want to bind for connect() */ -#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY) +#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT) struct sockaddr_storage tproxy_addr; /* non-local address we want to bind to for connect() */ char *bind_hdr_name; /* bind to
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Fri, Apr 26, 2013 at 12:55:23AM +0200, PiBa-NL wrote: Hi All / Developers, Seams i have tranparent proxying working now on FreeBSD 8.3 with HAProxy1.5dev18 + small modification. Needed to add a firewall forwarding rule to forward the traffic to the localhost for socket processing. Could a developer please make the following change? /*/* Add the following on line 33 of /include/common/compact.h */*//* *//*#ifdef *//*IP_BINDANY*//* *//* /* FreeBSD define variables */*//* *//* #define SOL_IP IPPROTO_IP*//* *//* #define SOL_IPV6 IPPROTO_IPV6*//* *//* #define IP_TRANSPARENT IP_BINDANY*//* *//*#endif*/ It's quite hard to exactly understand what needs to be changed with such a syntax, could you please send a standard patch ? For this, just do a diff -urN between the original source directory and the modified one. Also I'm wondering whether we should define USE_FREEBSD_TPROXY instead of USE_LINUX_TPROXY for this. Maybe we should rename CONFIG_HAP_LINUX_TPROXY to CONFIG_HAP_FULL_TPROXY and adapt it depending on the OS. After this haproxy can be successfully compiled on FreeBSD8.3 with the USE_LINUX_TPROXY=yes build option and transparent proxying works when the fwd firewall rule is made active. On my pfSense2.1 system the following worked to load ipfw and add the fwd rule in ipfw.. : /sbin/kldload ipfw /sbin/sysctl net.inet.ip.pfil.inbound=pf net.inet6.ip6.pfil.inbound=pf net.inet.ip.pfil.outbound=pf net.inet6.ip6.pfil.outbound=pf /sbin/sysctl net.link.ether.ipfw=1 ipfw_context -a haproxy ipfw_context -s haproxy ipfw_context -a haproxy -n em0 *ipfw -x haproxy add 20 fwd localhost tcp from IP-BACKEND-SERVER 80 to any in recv em0* (This firewall rule should actually also check if the correct 'uid' of the haproxy process is set to also allow directly contacting the backendserver, but i could not get that part to work though that is not a HAProxy issue so should get fixed elsewhere.) And ideally it should be possible with 'pf' instead of 'ipfw', but that is still something i'm trying to investigate.. Maybe such information should go into a dedicated file in the doc/ directory. If this is not the correct way to fix/change this for FreeBSD could someone please advice on what is.? Thanks in advance. PiBa-NL Thanks, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy, Sorry for the weird syntax.. I made the text 'bold', but that seams to have come out differently... Anyway i hope the 'patch' below is something you can work with.? As for renaming the CONFIG_HAP_LINUX_TPROXY to something different would require everyone that on a regular basis builds HAProxy with this feature to change their build flags.. So i don't think it should be renamed/removed. Also while adding another flag for clarity purposes, i don't think it really adds that much ease of use, and would require new make scripts and several other changes though-out where transparent proxying is implemented. Ive changed the defines a little more to i think be 'best compatible' with i think any circumstances.. Is this something that you could 'apply'?: --- workoriginal/haproxy-1.5-dev18/include/common/compat.h 2013-04-26 19:36:15.0 + +++ work/haproxy-1.5-dev18/include/common/compat.h 2013-04-26 20:32:15.0 + @@ -81,7 +81,16 @@ #endif /* On Linux, IP_TRANSPARENT and/or IP_FREEBIND generally require a kernel patch */ +/* On FreeBSD, IP_BINDANY is supported from FreeBSD 8 and up */ #if defined(CONFIG_HAP_LINUX_TPROXY) + #if defined(BSD) defined(IP_BINDANY) defined(IPV6_BINDANY) +/* FreeBSD defines */ +#define SOL_IP IPPROTO_IP +#define SOL_IPV6 IPPROTO_IPV6 +#define IP_TRANSPARENT IP_BINDANY +#define IPV6_TRANSPARENT IPV6_BINDANY + #endif + #if !defined(IP_FREEBIND) #define IP_FREEBIND 15 #endif /* !IP_FREEBIND */ Op 26-4-2013 8:33, Willy Tarreau schreef: Hi, On Fri, Apr 26, 2013 at 12:55:23AM +0200, PiBa-NL wrote: Hi All / Developers, Seams i have tranparent proxying working now on FreeBSD 8.3 with HAProxy1.5dev18 + small modification. Needed to add a firewall forwarding rule to forward the traffic to the localhost for socket processing. Could a developer please make the following change? /*/* Add the following on line 33 of /include/common/compact.h */*//* *//*#ifdef *//*IP_BINDANY*//* *//* /* FreeBSD define variables */*//* *//* #define SOL_IP IPPROTO_IP*//* *//* #define SOL_IPV6 IPPROTO_IPV6*//* *//* #define IP_TRANSPARENT IP_BINDANY*//* *//*#endif*/ It's quite hard to exactly understand what needs to be changed with such a syntax, could you please send a standard patch ? For this, just do a diff -urN between the original source directory and the modified one. Also I'm wondering whether we should define USE_FREEBSD_TPROXY instead of USE_LINUX_TPROXY for this. Maybe we should rename CONFIG_HAP_LINUX_TPROXY to CONFIG_HAP_FULL_TPROXY and adapt it depending on the OS. After this haproxy can be successfully compiled on FreeBSD8.3 with the USE_LINUX_TPROXY=yes build option and transparent proxying works when the fwd firewall rule is made active. On my pfSense2.1 system the following worked to load ipfw and add the fwd rule in ipfw.. : /sbin/kldload ipfw /sbin/sysctl net.inet.ip.pfil.inbound=pf net.inet6.ip6.pfil.inbound=pf net.inet.ip.pfil.outbound=pf net.inet6.ip6.pfil.outbound=pf /sbin/sysctl net.link.ether.ipfw=1 ipfw_context -a haproxy ipfw_context -s haproxy ipfw_context -a haproxy -n em0 *ipfw -x haproxy add 20 fwd localhost tcp from IP-BACKEND-SERVER 80 to any in recv em0* (This firewall rule should actually also check if the correct 'uid' of the haproxy process is set to also allow directly contacting the backendserver, but i could not get that part to work though that is not a HAProxy issue so should get fixed elsewhere.) And ideally it should be possible with 'pf' instead of 'ipfw', but that is still something i'm trying to investigate.. Maybe such information should go into a dedicated file in the doc/ directory. If this is not the correct way to fix/change this for FreeBSD could someone please advice on what is.? Thanks in advance. PiBa-NL Thanks, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Fri, Apr 26, 2013 at 08:40:49PM +0200, PiBa-NL wrote: Hi Willy, Sorry for the weird syntax.. I made the text 'bold', but that seams to have come out differently... As you can guess, there is no bold attribute in source code, so it must necessarily appear differently at some point. Anyway i hope the 'patch' below is something you can work with.? As for renaming the CONFIG_HAP_LINUX_TPROXY to something different would require everyone that on a regular basis builds HAProxy with this feature to change their build flags.. So i don't think it should be renamed/removed. Also while adding another flag for clarity purposes, i don't think it really adds that much ease of use, and would require new make scripts and several other changes though-out where transparent proxying is implemented. I think it would not cause real issues, unless people are forcing all CFLAGS, which would be a bit hasardous, but still possible. So I think we'll stick to not changing it. Ive changed the defines a little more to i think be 'best compatible' with i think any circumstances.. Is this something that you could 'apply'?: Yes, definitely, thank you ! Well, your mailer seems to have wrapped lines, but they're just a few so I can fix that by hand : --- workoriginal/haproxy-1.5-dev18/include/common/compat.h 2013-04-26 19:36:15.0 + +++ work/haproxy-1.5-dev18/include/common/compat.h 2013-04-26 20:32:15.0 + @@ -81,7 +81,16 @@ #endif /* On Linux, IP_TRANSPARENT and/or IP_FREEBIND generally require a kernel patch */ +/* On FreeBSD, IP_BINDANY is supported from FreeBSD 8 and up */ #if defined(CONFIG_HAP_LINUX_TPROXY) + #if defined(BSD) defined(IP_BINDANY) defined(IPV6_BINDANY) +/* FreeBSD defines */ +#define SOL_IP IPPROTO_IP +#define SOL_IPV6 IPPROTO_IPV6 +#define IP_TRANSPARENT IP_BINDANY +#define IPV6_TRANSPARENT IPV6_BINDANY + #endif + #if !defined(IP_FREEBIND) #define IP_FREEBIND 15 #endif /* !IP_FREEBIND */ I'll add a comment in the README about this option being available in FreeBSD 8+ as well. If you could add a small file with your config example, it would be nice, so that people don't ask how to do it all the time on the mailing list. Thanks, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Now that I'm seeing the patch in its context, I think it's not the cleanest way to do it, because we redefine IPV6_TRANSPARENT and IP_TRANSPARENT when IP_FREEBIND is not defined, while we don't use this one and the other ones are defined below. Also I'm concerned about the result of running this with an unsupported kernel, which will fail the IP_TRANSPARENT syscall and fall back to IP_FREEBIND, arbitrarily forced to 15 (since it's for Linux) and which I don't know what it means on FreeBSD. So I'm thinking again about having a separate USE_FREEBSD_TPROXY option, I'll see what I can come up with. Regards, Willy
RE: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, throwing in my two cents here, based on a few uneducated guesses reading the Makefile, etc. Feel free to disagree/correct/shout at me :) (actually I wrote this before Willy answered) As for renaming the CONFIG_HAP_LINUX_TPROXY to something different would require everyone that on a regular basis builds HAProxy with this feature to change their build flags.. The name CONFIG_HAP_LINUX_TPROXY or USE_LINUX_TPROXY suggests this is for Linux. Implementing compatibility changes for FreeBSD in those flags is misleading, whether they are internal (like CONFIG_HAP_LINUX_TPROXY) or external (USE_LINUX_TPROXY). I think we should avoid that. would require new make scripts and several other changes though-out where transparent proxying is implemented. If you built haproxy before on FreeBSD and used transparent proxying, then it probably didn't work at all (like in your case) or those values have been defined in another place, like by libc or by a manual definition. Either way, we don't break anything that currently works by introducing a new build flag. So you would only have to adjust your make line if you actually need it and I think thats does less long-term harm than defining those things under the USE_LINUX_TPROXY/CONFIG_HAP_LINUX_TPROXY hat. Also I'm wondering whether we should define USE_FREEBSD_TPROXY instead of USE_LINUX_TPROXY for this. Maybe we should rename CONFIG_HAP_LINUX_TPROXY to CONFIG_HAP_FULL_TPROXY and adapt it depending on the OS. Yes, that makes more sense to me. We should probably clarify the condition with OpenBSD. I assume those defines are the same for all BSD flavors? So should we introduce a more generic flag like USE_BSD_TPROXY instead to avoid a USE flag for every BSD or does the difference between them justify a per-BSD USE flag? Regards, Lukas
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Lukas, On Fri, Apr 26, 2013 at 10:26:33PM +0200, Lukas Tribus wrote: Hi, throwing in my two cents here, based on a few uneducated guesses reading the Makefile, etc. Feel free to disagree/correct/shout at me :) Thanks for sharing your thoughts, I feel less alone sometimes when I can discuss choices. (actually I wrote this before Willy answered) As for renaming the CONFIG_HAP_LINUX_TPROXY to something different would require everyone that on a regular basis builds HAProxy with this feature to change their build flags.. The name CONFIG_HAP_LINUX_TPROXY or USE_LINUX_TPROXY suggests this is for Linux. Implementing compatibility changes for FreeBSD in those flags is misleading, whether they are internal (like CONFIG_HAP_LINUX_TPROXY) or external (USE_LINUX_TPROXY). I think we should avoid that. OK, that was my first impression as well. It looks unpolished but can be understood sometimes during a transition, but not for the final state. would require new make scripts and several other changes though-out where transparent proxying is implemented. If you built haproxy before on FreeBSD and used transparent proxying, then it probably didn't work at all (like in your case) or those values have been defined in another place, like by libc or by a manual definition. Either way, we don't break anything that currently works by introducing a new build flag. So you would only have to adjust your make line if you actually need it and I think thats does less long-term harm than defining those things under the USE_LINUX_TPROXY/CONFIG_HAP_LINUX_TPROXY hat. Also I'm wondering whether we should define USE_FREEBSD_TPROXY instead of USE_LINUX_TPROXY for this. Maybe we should rename CONFIG_HAP_LINUX_TPROXY to CONFIG_HAP_FULL_TPROXY and adapt it depending on the OS. Yes, that makes more sense to me. We should probably clarify the condition with OpenBSD. I assume those defines are the same for all BSD flavors? So should we introduce a more generic flag like USE_BSD_TPROXY instead to avoid a USE flag for every BSD or does the difference between them justify a per-BSD USE flag? OK I have reviewed the existing code a bit. Most of the usages of CONFIG_HAP_LINUX_TPROXY are : 1) define the appropriate flags when they're not defined (kernel newer than libc) 2) enable parsing the option 3) enable the setsockopt calls (one of which is wrong for FBSD). So what I'm thinking about is to change that : 1) solely rely on the various per-OS flags to decide whether or not transparent proxy is supported (eg: IP_FREEBIND, IP_TRANSPARENT, IP_BINDANY, ...). That way we don't need an OS-specific option for something that automatically comes with the OS and that can be detected using a #ifdef and is enabled using a config setting anyway (eg: the transparent or usesrc keyword). 2) keep CONFIG_HAP_LINUX_TPROXY to force setting the values on linux when they're not set (as it initially was made for) 3) only implement the setsockopt() that have their appropriate define. 4) report in the -vv output what options are supported. Thus it will become trivial to add support for other OSes (I believe OpenBSD also supports it). What do you think about this ? Regards, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy / Lukas, It seams to me OpenBSD doesn't support the IP_BINDANY flag..: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/netinet/in.h http://www.openbsd.org/cgi-bin/cvsweb/%7Echeckout%7E/src/sys/netinet/in.h While FreeBSD does: http://svnweb.freebsd.org/base/head/sys/netinet/in.h?view=markup But then again neither of them supports SOL_IP, so i would expect compilation to simply 'fail'. When trying to compile with the USE_LINUX_TPROXY option. The combination i think is unlikely to cause problems for other currently working builds/systems.. If you want i can probably come up with a combination that makes it work for FreeBSD with a special USE_FREEBSD_TPROXY make option. Or go for the 'full automatic inclusion' depending on available flags. Which i think is even 'nicer'. But probably needs more testing to confirm proper working.. I would be willing to make these changes. Is this the way to go? Thanks for reviewing my proposed changes sofar. PiBa-NL Op 26-4-2013 22:40, Willy Tarreau schreef: Hi Lukas, On Fri, Apr 26, 2013 at 10:26:33PM +0200, Lukas Tribus wrote: Hi, throwing in my two cents here, based on a few uneducated guesses reading the Makefile, etc. Feel free to disagree/correct/shout at me :) Thanks for sharing your thoughts, I feel less alone sometimes when I can discuss choices. (actually I wrote this before Willy answered) As for renaming the CONFIG_HAP_LINUX_TPROXY to something different would require everyone that on a regular basis builds HAProxy with this feature to change their build flags.. The name CONFIG_HAP_LINUX_TPROXY or USE_LINUX_TPROXY suggests this is for Linux. Implementing compatibility changes for FreeBSD in those flags is misleading, whether they are internal (like CONFIG_HAP_LINUX_TPROXY) or external (USE_LINUX_TPROXY). I think we should avoid that. OK, that was my first impression as well. It looks unpolished but can be understood sometimes during a transition, but not for the final state. would require new make scripts and several other changes though-out where transparent proxying is implemented. If you built haproxy before on FreeBSD and used transparent proxying, then it probably didn't work at all (like in your case) or those values have been defined in another place, like by libc or by a manual definition. Either way, we don't break anything that currently works by introducing a new build flag. So you would only have to adjust your make line if you actually need it and I think thats does less long-term harm than defining those things under the USE_LINUX_TPROXY/CONFIG_HAP_LINUX_TPROXY hat. Also I'm wondering whether we should define USE_FREEBSD_TPROXY instead of USE_LINUX_TPROXY for this. Maybe we should rename CONFIG_HAP_LINUX_TPROXY to CONFIG_HAP_FULL_TPROXY and adapt it depending on the OS. Yes, that makes more sense to me. We should probably clarify the condition with OpenBSD. I assume those defines are the same for all BSD flavors? So should we introduce a more generic flag like USE_BSD_TPROXY instead to avoid a USE flag for every BSD or does the difference between them justify a per-BSD USE flag? OK I have reviewed the existing code a bit. Most of the usages of CONFIG_HAP_LINUX_TPROXY are : 1) define the appropriate flags when they're not defined (kernel newer than libc) 2) enable parsing the option 3) enable the setsockopt calls (one of which is wrong for FBSD). So what I'm thinking about is to change that : 1) solely rely on the various per-OS flags to decide whether or not transparent proxy is supported (eg: IP_FREEBIND, IP_TRANSPARENT, IP_BINDANY, ...). That way we don't need an OS-specific option for something that automatically comes with the OS and that can be detected using a #ifdef and is enabled using a config setting anyway (eg: the transparent or usesrc keyword). 2) keep CONFIG_HAP_LINUX_TPROXY to force setting the values on linux when they're not set (as it initially was made for) 3) only implement the setsockopt() that have their appropriate define. 4) report in the -vv output what options are supported. Thus it will become trivial to add support for other OSes (I believe OpenBSD also supports it). What do you think about this ? Regards, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
On Fri, Apr 26, 2013 at 11:03:00PM +0200, PiBa-NL wrote: Hi Willy / Lukas, It seams to me OpenBSD doesn't support the IP_BINDANY flag..: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/netinet/in.h http://www.openbsd.org/cgi-bin/cvsweb/%7Echeckout%7E/src/sys/netinet/in.h it seems it has, but differently : http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-07/msg00399.html While FreeBSD does: http://svnweb.freebsd.org/base/head/sys/netinet/in.h?view=markup But then again neither of them supports SOL_IP, so i would expect compilation to simply 'fail'. When trying to compile with the USE_LINUX_TPROXY option. Which is exactly the reason I don't want to remap these things which are linux-specific, and instead use the proper call depending on the available flags. Eg something like this : #if defined(SOL_IP) defined(IP_TRANSPARENT) /* linux */ ret = setsockop(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* freebsd */ ret = setsockop(fd, IP_PROTOIP, IP_BINDANY, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* openbsd */ ret = setsockop(fd, SOL_SOCKET, SO_BINDANY, one, sizeof(one)); #else /* unsupported platform */ ret = -1; #endif The combination i think is unlikely to cause problems for other currently working builds/systems.. If you want i can probably come up with a combination that makes it work for FreeBSD with a special USE_FREEBSD_TPROXY make option. No, really I think something like above is much better for the long term. It's more work to adapt existing code first but will pay in the long term, even in the short term if it allows us to support OpenBSD at the same time. Or go for the 'full automatic inclusion' depending on available flags. Which i think is even 'nicer'. But probably needs more testing to confirm proper working.. I would be willing to make these changes. Is this the way to go? As you like, if you feel comfortable with changing the way the current code works (the linux-specific one), feel free to try, otherwise I can do it over the week-end, and then a second patch derived from yours will bring in support for FreeBSD then OpenBSD if someone here is able to test it. Thanks for reviewing my proposed changes sofar. you're welcome :-) Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy, Ill give it a try and send the patch as an attachment, though im not 100% comfortable with the code. I think i can do it. Will take me a few days though.. Thanks sofar. Op 26-4-2013 23:12, Willy Tarreau schreef: On Fri, Apr 26, 2013 at 11:03:00PM +0200, PiBa-NL wrote: Hi Willy / Lukas, It seams to me OpenBSD doesn't support the IP_BINDANY flag..: http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/sys/netinet/in.h http://www.openbsd.org/cgi-bin/cvsweb/%7Echeckout%7E/src/sys/netinet/in.h it seems it has, but differently : http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-07/msg00399.html While FreeBSD does: http://svnweb.freebsd.org/base/head/sys/netinet/in.h?view=markup But then again neither of them supports SOL_IP, so i would expect compilation to simply 'fail'. When trying to compile with the USE_LINUX_TPROXY option. Which is exactly the reason I don't want to remap these things which are linux-specific, and instead use the proper call depending on the available flags. Eg something like this : #if defined(SOL_IP) defined(IP_TRANSPARENT) /* linux */ ret = setsockop(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* freebsd */ ret = setsockop(fd, IP_PROTOIP, IP_BINDANY, one, sizeof(one)); #elif defined (IP_PROTOIP) defined(IP_BINDANY) /* openbsd */ ret = setsockop(fd, SOL_SOCKET, SO_BINDANY, one, sizeof(one)); #else /* unsupported platform */ ret = -1; #endif The combination i think is unlikely to cause problems for other currently working builds/systems.. If you want i can probably come up with a combination that makes it work for FreeBSD with a special USE_FREEBSD_TPROXY make option. No, really I think something like above is much better for the long term. It's more work to adapt existing code first but will pay in the long term, even in the short term if it allows us to support OpenBSD at the same time. Or go for the 'full automatic inclusion' depending on available flags. Which i think is even 'nicer'. But probably needs more testing to confirm proper working.. I would be willing to make these changes. Is this the way to go? As you like, if you feel comfortable with changing the way the current code works (the linux-specific one), feel free to try, otherwise I can do it over the week-end, and then a second patch derived from yours will bring in support for FreeBSD then OpenBSD if someone here is able to test it. Thanks for reviewing my proposed changes sofar. you're welcome :-) Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi All / Baptiste,
It seams i have found error in my initial email sorry for that. It
should have mentioned IP_BINDANY in the example line of code instead of
IP_FREEBIND. Still the my problem remains the same.
Returning SYN-ACK packets are not recieved/intercepted by HAProxy. And
the browser gets a No server is available to handle this request.
after exactly 2 minutes.
There seam to be needed at least two changes for compiling/using the
full-transparent-proxy feature off FreeBSD.
In the file *compat.h* (or anywhere that fits better.) add:
/*#ifndef SOL_IP**
**#define SOL_IP 0**
**#define SOL_IPV6 0**
**#endif*/*
*
In the file *proto_tcp.c* change:
/if (flags ip_transp_working) {//
//if (//
///*#ifdef **/*IP_BINDANY*/**
*/ //*setsockopt(fd, SOL_IP, IP_BINDANY, one, sizeof(one))*//*|| *//
*#endif*/*
* /setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)) == 0//
//|| setsockopt(fd, SOL_IP, IP_FREEBIND, one,
sizeof(one)) == 0)//
//{//
//foreign_ok = 1;//
//}//
//else//
//ip_transp_working = 0;//
//}/
This makes it work for the sending a SYN connection request to the
backend server... However the SYN-ACK response, though it is passing
through the HAProxy server, is not recieved by HAProxy.
A tcpdump with mac-addresses on the LAN side of the FreeBSD router shows
shows that .1.50 contacts haproxy on .1.22:81, gets a SYN-ACK reply
back, but after that also SYN-ACK packets from the backend server.. as
can be seen from the MAC adresses all recieved traffic comes through the
FreeBSD machine:
21:03:06.018602 00:0c:29:0d:89:90 00:0c:29:b5:f0:fe, ethertype IPv4
(0x0800), length 62: 192.168.1.50.1177 192.168.1.22.81: Flags [S], seq
252352307, win 64240, options [mss 1460,nop,nop,sackOK], length 0
21:03:06.018735 00:0c:29:b5:f0:fe 00:0c:29:0d:89:90, ethertype IPv4
(0x0800), length 62: 192.168.1.22.81 192.168.1.50.1177: Flags [S.],
seq 2295999072, ack 252352308, win 65228, options [mss 1460,sackOK,eol],
length 0
21:03:06.019066 00:0c:29:0d:89:90 00:0c:29:b5:f0:fe, ethertype IPv4
(0x0800), length 60: 192.168.1.50.1177 192.168.1.22.81: Flags [.], ack
1, win 64240, length 0
21:03:06.020429 00:0c:29:0d:89:90 00:0c:29:b5:f0:fe, ethertype IPv4
(0x0800), length 336: 192.168.1.50.1177 192.168.1.22.81: Flags [P.],
ack 1, win 64240, length 282
21:03:06.020517 00:0c:29:b5:f0:fe 00:0c:29:0d:89:90, ethertype IPv4
(0x0800), length 54: 192.168.1.22.81 192.168.1.50.1177: Flags [.], ack
283, win 65418, length 0
21:03:06.021205 00:0c:29:b5:f0:fe 00:0c:29:0d:89:90, ethertype IPv4
(0x0800), length 74: 192.168.0.40.81 192.168.1.50.49496: Flags [S.],
seq 2271887215, ack 4020735996, win 8192, options [mss 1260,nop,wscale
8,sackOK,TS val 144738 ecr 92963], length 0
21:03:09.026308 00:0c:29:b5:f0:fe 00:0c:29:0d:89:90, ethertype IPv4
(0x0800), length 74: 192.168.0.40.81 192.168.1.50.49496: Flags [S.],
seq 2271887215, ack 4020735996, win 8192, options [mss 1260,nop,wscale
8,sackOK,TS val 145038 ecr 92963], length 0
21:03:15.025375 00:0c:29:b5:f0:fe 00:0c:29:0d:89:90, ethertype IPv4
(0x0800), length 70: 192.168.0.40.81 192.168.1.50.49496: Flags [S.],
seq 2271887215, ack 4020735996, win 8192, options [mss 1260,sackOK,TS
val 145638 ecr 92963], length 0
Those last three packets should not go towards the browser pc..
Op 17-4-2013 21:55, PiBa-NL schreef:
Hi Baptiste,
Thanks for your reply, i understand that the traffic must pass through
the router/HAProxy box, and for that part i can confirm that the
routing of packets is 'working' like it should.
To explain my setup a little more:
*Webserver* (Win7) 192.168.0.40/24 on a OPT1/DMZ using gateway
:192.168.0.117
*HAProxy* on FreeBSD 8.3(pfSense2.1) which performs routing between
the 2 networks has 2 interfaces(that matter):
DMZ interface has 192.168.0.117/24
LAN interface has 192.168.1.1/24
*ClientPC* (WinXP) 192.168.1.50/24 on the LAN gateway :192.168.1.1
(p.s. to make the picture complete and a little more complicated the
FreeBSD and ClientPC are running within 'VMware Workstation' which
runs on the Win7 machine., But im positive the networks are correctly
separated (using a 'LAN-segment' between FreeBSD and the ClientPC) and
traffic does flow through the FreeBSD machine.)
All traffic moving between clientPC and the Webserver 'must' go
through the FreeBSD machine which routes the traffic, i have confirmed
that traffic moves correctly through the FreeBSD machine by using
wireshark and tcpdump on all machines.
So i have traced the packets of the connection to go like this:
1-ClientPC browser 'connects' to HAProxy and waits for the webpage (ok)
2-HAProxy sends SYN packet to Webserver with a spoofed source IP (ok)
3-Webserver sends response SYN-ack back to the spoofed source IP to
its gateway 192.168.0.117 (and the MAC address using that IP) (expected)
4-FreeBSD passes the SYN-ack on to the
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
I forgot to mention im using HAproxy 1.5dev18. Hello HAProxy developers/users, I would like to be able to run HAProxy transparently on FreeBSD 8.3. This would be both for my own usage and also to make it available to a larger public by including it in a 'haproxy-devel' package for pfSense. However when trying to use it i get the error: /[ALERT] 104/235847 (72477) : parsing [/var/etc/haproxy.cfg:34] : 'usesrc' not allowed here because support for TPROXY was not compiled in./ From what i read it seams it should be possible. For example the Makefile contains the following: /ifeq ($(TARGET),freebsd)// USE_TPROXY = implicit/ Which seams like it is supposed to be 'supported'. Ive also tried the USE_LINUX_TPROXY=yes compile flag, but this returns 2 undeclared variables *SOL_IP* and *SOL_IPV6*. Ive tried declaring them with substitute values like 'IP_BINDANY', or the value 6 which could stand for the TCP protocol type, or 0. , but though the source did then compile the end result still was that either an error was returned to the browser that no backend was available, together with the following debug error: *[ALERT] 104/235129 (17380) : Cannot bind to tproxy source address before connect() for backend pb3TEST_http. Aborting.** *Or i dont get a response at all and HAproxy seems to be waiting for 'something' to happen.. Could it be that something is not fully supported in HAProxy toghether with FreeBSD to allow transparent proxying? Or am i looking at the wrong side of the problem and would i need to compile the FreeBSD kernel with tproxy support.? Which I believe would be natively supported in version 8, but i might be wrong on that.. I i add after /*setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one)*/ this line: */setsockopt(fd, SOL_IP, IP_FREEBIND, one, sizeof(one));/* It removes the error about 'Cannot bind to tproxy source address...' and packets do seam to be send to the proper destination. Except the connection never establishes.. The browser running on 192.168.1.50 contacts haproxy on its IP:port http://192.168.1.22:81/ Haproxy then forwards the traffic to the server 192.168.0.40.81 which is according to status page L7OK/200 in 0ms. Also the reply packets gets routed back to the original client pc (wireshark confirmed that..), and seam not to get intercepted by HAproxy which i think is supposed to happen.?.. when passing through the 'FreeBSD router'. But when performing a tcpdump on the interface in the 192.168.0.117 network only SYN and SYN-ack packets seem to be 'exchanged'.. Bet never any actual 'data' 21:02:04.915310 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:04.915464 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281557 ecr 1556876], length 0 21:02:04.915546 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:07.910690 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281857 ecr 1556876], length 0 21:02:07.911073 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:07.911079 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:11.110673 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:11.110685 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:13.913959 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,sackOK,TS val 1282457 ecr 1556876], length 0 While when the pc contacts the webserver directly it works 'normally'.. 20:45:12.746359 IP 192.168.1.50.3588 192.168.0.40.81: Flags [S], seq 24635592, win 64240, options [mss 1460,nop,nop,sackOK], length 0 20:45:12.746473 IP 192.168.0.40.81 192.168.1.50.3588: Flags [S.], seq 3353931105, ack 24635593, win 8192, options [mss 1260,nop,nop,sackOK], length 0 20:45:12.746937 IP 192.168.1.50.3588 192.168.0.40.81: Flags [.], ack 1, win 64260, length 0 20:45:12.747071 IP 192.168.1.50.3588 192.168.0.40.81: Flags [P.], ack 1, win 64260, length 282 20:45:12.750878 IP 192.168.0.40.81 192.168.1.50.3588: Flags [.], ack 283, win 65520, length 1260 20:45:12.751005 IP 192.168.0.40.81 192.168.1.50.3588: Flags [P.], ack 283, win 65520, length 1142 20:45:12.751463 IP 192.168.1.50.3588 192.168.0.40.81: Flags [.], ack 2403, win 64260, length 0
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, In order to work in transparent proxy mode, the server's default gateway must be the HAproxy server. Or at least, the traffic from the server must pass through the haproxy box before reaching the client. Even if HAProxy spoof the client IP, it's HAProxy which initializes the TCP connection, so if the server try to reach the client directly, this one would refuse the connection. Baptiste On Wed, Apr 17, 2013 at 8:01 PM, PiBa-NL piba.nl@gmail.com wrote: I forgot to mention im using HAproxy 1.5dev18. Hello HAProxy developers/users, I would like to be able to run HAProxy transparently on FreeBSD 8.3. This would be both for my own usage and also to make it available to a larger public by including it in a 'haproxy-devel' package for pfSense. However when trying to use it i get the error: [ALERT] 104/235847 (72477) : parsing [/var/etc/haproxy.cfg:34] : 'usesrc' not allowed here because support for TPROXY was not compiled in. From what i read it seams it should be possible. For example the Makefile contains the following: ifeq ($(TARGET),freebsd) USE_TPROXY = implicit Which seams like it is supposed to be 'supported'. Ive also tried the USE_LINUX_TPROXY=yes compile flag, but this returns 2 undeclared variables SOL_IP and SOL_IPV6. Ive tried declaring them with substitute values like 'IP_BINDANY', or the value 6 which could stand for the TCP protocol type, or 0. , but though the source did then compile the end result still was that either an error was returned to the browser that no backend was available, together with the following debug error: [ALERT] 104/235129 (17380) : Cannot bind to tproxy source address before connect() for backend pb3TEST_http. Aborting. Or i dont get a response at all and HAproxy seems to be waiting for 'something' to happen.. Could it be that something is not fully supported in HAProxy toghether with FreeBSD to allow transparent proxying? Or am i looking at the wrong side of the problem and would i need to compile the FreeBSD kernel with tproxy support.? Which I believe would be natively supported in version 8, but i might be wrong on that.. I i add after setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one) this line: setsockopt(fd, SOL_IP, IP_FREEBIND, one, sizeof(one)); It removes the error about 'Cannot bind to tproxy source address...' and packets do seam to be send to the proper destination. Except the connection never establishes.. The browser running on 192.168.1.50 contacts haproxy on its IP:port http://192.168.1.22:81/ Haproxy then forwards the traffic to the server 192.168.0.40.81 which is according to status page L7OK/200 in 0ms. Also the reply packets gets routed back to the original client pc (wireshark confirmed that..), and seam not to get intercepted by HAproxy which i think is supposed to happen.?.. when passing through the 'FreeBSD router'. But when performing a tcpdump on the interface in the 192.168.0.117 network only SYN and SYN-ack packets seem to be 'exchanged'.. Bet never any actual 'data' 21:02:04.915310 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:04.915464 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281557 ecr 1556876], length 0 21:02:04.915546 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1556876 ecr 0], length 0 21:02:07.910690 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,nop,wscale 8,sackOK,TS val 1281857 ecr 1556876], length 0 21:02:07.911073 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:07.911079 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557176 ecr 0], length 0 21:02:11.110673 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:11.110685 IP 192.168.1.50.51194 192.168.0.40.81: Flags [S], seq 352103919, win 65228, options [mss 1460,nop,wscale 7,sackOK,TS val 1557496 ecr 0], length 0 21:02:13.913959 IP 192.168.0.40.81 192.168.1.50.51194: Flags [S.], seq 4102632929, ack 352103920, win 8192, options [mss 1260,sackOK,TS val 1282457 ecr 1556876], length 0 While when the pc contacts the webserver directly it works 'normally'.. 20:45:12.746359 IP 192.168.1.50.3588 192.168.0.40.81: Flags [S], seq 24635592, win 64240, options [mss 1460,nop,nop,sackOK], length 0 20:45:12.746473 IP 192.168.0.40.81 192.168.1.50.3588: Flags [S.], seq 3353931105, ack 24635593, win 8192, options [mss 1260,nop,nop,sackOK], length 0 20:45:12.746937 IP
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Baptiste, Thanks for your reply, i understand that the traffic must pass through the router/HAProxy box, and for that part i can confirm that the routing of packets is 'working' like it should. To explain my setup a little more: *Webserver* (Win7) 192.168.0.40/24 on a OPT1/DMZ using gateway :192.168.0.117 *HAProxy* on FreeBSD 8.3(pfSense2.1) which performs routing between the 2 networks has 2 interfaces(that matter): DMZ interface has 192.168.0.117/24 LAN interface has 192.168.1.1/24 *ClientPC* (WinXP) 192.168.1.50/24 on the LAN gateway :192.168.1.1 (p.s. to make the picture complete and a little more complicated the FreeBSD and ClientPC are running within 'VMware Workstation' which runs on the Win7 machine., But im positive the networks are correctly separated (using a 'LAN-segment' between FreeBSD and the ClientPC) and traffic does flow through the FreeBSD machine.) All traffic moving between clientPC and the Webserver 'must' go through the FreeBSD machine which routes the traffic, i have confirmed that traffic moves correctly through the FreeBSD machine by using wireshark and tcpdump on all machines. So i have traced the packets of the connection to go like this: 1-ClientPC browser 'connects' to HAProxy and waits for the webpage (ok) 2-HAProxy sends SYN packet to Webserver with a spoofed source IP (ok) 3-Webserver sends response SYN-ack back to the spoofed source IP to its gateway 192.168.0.117 (and the MAC address using that IP) (expected) 4-FreeBSD passes the SYN-ack on to the ClientPC.(this is what should not happen.) ## HAproxy waits and retries for a few timeouts to occur.. and then eventually after about 2 minutes the browser is served the 503 page No server is available to handle this request. ## So to me it seams HAproxy should in step 4 somehow get the SYN-ack packet, im not sure if it should listen/bind on all local IP's or if the traffic should get passed back to a HAProxy port by using for example a NAT rule.?. PiBa-NL Op 17-4-2013 21:21, Baptiste schreef: Hi, In order to work in transparent proxy mode, the server's default gateway must be the HAproxy server. Or at least, the traffic from the server must pass through the haproxy box before reaching the client. Even if HAProxy spoof the client IP, it's HAProxy which initializes the TCP connection, so if the server try to reach the client directly, this one would refuse the connection. Baptiste On Wed, Apr 17, 2013 at 8:01 PM, PiBa-NL piba.nl@gmail.com wrote: I forgot to mention im using HAproxy 1.5dev18. Hello HAProxy developers/users, I would like to be able to run HAProxy transparently on FreeBSD 8.3. This would be both for my own usage and also to make it available to a larger public by including it in a 'haproxy-devel' package for pfSense. However when trying to use it i get the error: [ALERT] 104/235847 (72477) : parsing [/var/etc/haproxy.cfg:34] : 'usesrc' not allowed here because support for TPROXY was not compiled in. From what i read it seams it should be possible. For example the Makefile contains the following: ifeq ($(TARGET),freebsd) USE_TPROXY = implicit Which seams like it is supposed to be 'supported'. Ive also tried the USE_LINUX_TPROXY=yes compile flag, but this returns 2 undeclared variables SOL_IP and SOL_IPV6. Ive tried declaring them with substitute values like 'IP_BINDANY', or the value 6 which could stand for the TCP protocol type, or 0. , but though the source did then compile the end result still was that either an error was returned to the browser that no backend was available, together with the following debug error: [ALERT] 104/235129 (17380) : Cannot bind to tproxy source address before connect() for backend pb3TEST_http. Aborting. Or i dont get a response at all and HAproxy seems to be waiting for 'something' to happen.. Could it be that something is not fully supported in HAProxy toghether with FreeBSD to allow transparent proxying? Or am i looking at the wrong side of the problem and would i need to compile the FreeBSD kernel with tproxy support.? Which I believe would be natively supported in version 8, but i might be wrong on that.. I i add after setsockopt(fd, SOL_IP, IP_TRANSPARENT, one, sizeof(one) this line: setsockopt(fd, SOL_IP, IP_FREEBIND, one, sizeof(one)); It removes the error about 'Cannot bind to tproxy source address...' and packets do seam to be send to the proper destination. Except the connection never establishes.. The browser running on 192.168.1.50 contacts haproxy on its IP:port http://192.168.1.22:81/ Haproxy then forwards the traffic to the server 192.168.0.40.81 which is according to status page L7OK/200 in 0ms. Also the reply packets gets routed back to the original client pc (wireshark confirmed that..), and seam not to get intercepted by HAproxy which i think is supposed to happen.?.. when passing through the 'FreeBSD router'. But when performing a tcpdump on the interface in the 192.168.0.117 network only
