I have setup and configured an OpenBSD 5.3 amd64 system as a router with
multiple WAN links. All IPv4 traffic except basic FTP seems to work fine
and is load-balanced with redundancy across both WAN connections.
The pseudo setup is:
dsl_if (fxp1) \
--------router -------- internal_if (fxp2)
cable_if (fxp0) /
The router has been up and running for about two weeks with no problems
yet. I've simulated a single WAN failure during the install window by
disconnecting the dsl interface (and I tested the cable interface as well,
separately). When the interface goes down I see that 'netstat -nrf
inet|grep default' marks the failed connection's gateway and unusable and
all traffic continues on the working interface. Once the failed connection
is restored the gateway is detected and marked usable, and all traffic
balanced between the two interfaces.
I am really happy with this setup, except that FTP fails.
To setup this Multi-wan system I configured five files, hostname.fxp0,
hostname.fxp1, pf.conf rc.conf.local and sysctl.conf.
### /etc/hostname.fxp0:
inet <cable_IP> <cable_netmask>
!route add -mpath default <cable_GW>
### /etc/hostname.fxp1:
inet <dsl_IP> <dsl_netmask>
!route add -mpath default <dsl_GW>
### /etc/sysctl.conf:
net.inet.ip.forwarding=1
net.inet.ip.multipath=1
### /etc/rc.conf.local:
ftpproxy_flags="-T ftp-proxy -D7 -vv"
### /etc/pf.conf (relevant to FTP parts only):
cable_if = "fxp0"
cable_gw = <cable_GW> # I used the actual IP for the cable gateway here
dsl_if = "fxp1"
dsl_gw = <dsl_GW> # I used the actual IP for the dsl gateway here
int_if = "fxp2"
ftpproxy = "8021"
anchor "ftp-proxy/*"
pass out on $cable_if from $dsl_if route-to ( $dsl_if $dsl_gw )
pass out on $dsl_if from $cable_if route-to ( $cable_if $cable_gw )
match out on $cable_if inet from !($cable_if) nat-to ($cable_if)
match out on $dsl_if inet from !($dsl_if) nat-to ($dsl_if)
### end of config files
Note that I am not doing anything with ifstated, nor am i using the
route-to option in pf.conf to balance outgoing traffic. It seems that the
kernel handles both of these functions just fine as is in 5.3.
Initially to allow FTP, I was doing something in pf.conf like:
pass in quick on $internal_if inet proto tcp to any port ftp divert-to
127.0.0.1 port $ftpproxy
pass out on $cable_if proto tcp from ($cable_if) to any port ftp
pass out on $dsl_if proto tcp from $dsl_if) to any port ftp
...and ftp-proxy was run without any flags.
This resulted in ~1/2 of all connections failing. From a command line ftp
client you could login to the ftp server, but once you go PASV to get a
directory listing the connection would fail. The next time you reconnect,
you could get a directory listing and continue your work until you
disconnected. The next time, your ls would fail. If you reconnected a 4th
time, everything would work.
I figured that tagging the ftp-proxy helps. I changed the ftp-proxy
program to run with the flags:
# ftp-proxy -T ftp-proxy
... and I set up pf.conf to do this instead:
pass in quick on $internal_if inet proto tcp to port ftp divert-to
127.0.0.1 port $ftpproxy
pass in quick on $cable_if tagged ftp-proxy reply-to ( $cable_if $cable_gw
)
pass out quick on $cable_if proto tcp to any port ftp
The result of this setup is that I can login to an ftp server, and when I
do ls I get:
$ ftp -a ftp2.us.freebsd.org
Connected to mirror.symnds.com.
220 Welcome to mirror.symnds.net
331 Please specify the password.
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
229 Entering Extended Passive Mode (|||50153|)
ftp: Can't connect to `63.245.196.124:50153': Connection refused
200 EPRT command successful. Consider using EPSV.
150 Here comes the directory listing.
lrwxrwxrwx 1 ftp ftp 20 Jan 13 08:04 CentOS ->
distributions/CentOS
lrwxrwxrwx 1 ftp ftp 21 Jan 30 2012 FreeBSD ->
distributions/freebsd
lrwxrwxrwx 1 ftp ftp 11 Jan 13 08:24 cm -> software/cm
lrwxrwxrwx 1 ftp ftp 20 Feb 14 2012 debian ->
distributions/debian
drwxrwsr-x 18 ftp ftp 4096 Apr 01 02:03 distributions
-rw-r--r-- 1 ftp ftp 1406 Jan 30 2012 favicon.ico
lrwxrwxrwx 1 ftp ftp 12 Sep 27 2012 pkg -> software/pkg
drwxr-xr-x 2 ftp ftp 4096 Jan 30 2012 pub
-rw-r--r-- 1 ftp ftp 524288 Jan 30 2012 smokeping.test
drwxr-xr-x 15 ftp ftp 4096 Apr 01 02:04 software
drwxr-xr-x 3 ftp ftp 4096 May 16 2012 stats
lrwxrwxrwx 1 ftp ftp 20 Jan 30 2012 ubuntu ->
distributions/ubuntu
226 Directory send OK.
Notice that I get both a connection refused, _and_ the directory listing.
This would be ok, except that browsers see the connection refused error and
give a "cannot connect to ftp server" page instead of the directory listing.
When I do 'pfctl -a ftp-proxy/<PID>.<#> -sr' I can see that the allocated
port number is not the same as the one mentioned by the FTP client above.
I hope I'm pretty close to having FTP working. If anyone has any
suggestions or ideas on how to get ftp-proxy to play well with this setup,
I'd be happy to try their ideas and report back.
Thanks!
Howard
--
--------------------------------------------------------------------------
Notice: This e-mail message, together with any attachments, contains
information of Lighthouse Instruments, LLC (Charlottesville, VA, USA) that
may be confidential, proprietary copyrighted and/or legally privileged, and
is intended solely for the use of the individual or entity named on this
message. If you are not the intended recipient, please return this by
e-mail and delete it.
