I am new to using PF, long time user of IPFilter. I switched to OpenbSD/PF last week to setup a system to provide me with redundancy for my outbound connections. The setup consists of 2 machines, each connected to a different internet connection, and serving two internal subnets. (See ASCII diagram below)
Each machine provides redundancy for the other. Either machine can be master for a subnet, but in normal operation, only one is master for a given subnet. The setup works great, my traffic from both subnets makes it to the internet even if one machine is down. My problem is with traffic between the two subnets. For example, I will start a ssh connection on one subnet, connecting to a machine on the other subnet and after a few sec, or it seems whenever a large amount of data goes through, the connection will stall/hang. I can do this will FTP, SSH, scp etc, even my mail hangs if I have more than say 5 messages waiting. I've tried to debug this myself with tcpdump, and I see the connections going through but when the app hangs, I do not really see anything out of the ordinary. My problem is exacerbated by the number of interfaces on each box, two for each subnet in this case, which makes watching traffic difficult. I've seen some postings about using route-to but I am not clear how that would help me here (or even what my true problem is). One test I was able to do was to take one machine offline, and test my resultant connectivity between the two subnets. I took one machine at a time offline, and found running with 1 server (regardless of which) I was able to transfer a large file with no problem at all between subnets. So I think I can rule out hardware as my issue. I have one running theory at this time, but I'm not exactly certain how I could go about testing it out. (viewing the ASCII diagram for this will help). Traffic entering on fxp0 on HOBBES (from 192.168.20.0/24) and destined for 192.168.30.0/24 has two possible routes. It could be routed to CALVIN ( the master for 192.168.30.0/24 ), or simply leave out fxp1 which is on the 30.0 network. Further, I have noticed through some testing that if I ping 192.168.30.1 from the 20.0 network, HOBBES responds, even though it is the backup for that carp interface. So packets never make it to the other server I believe. Now, the return traffic on the 30.0 network thinks (and rightly so) that it's default gateway is CALVIN, and if my above test is true, then instead of passing the packet over to HOBBES for routing to the 20.0 network, the packet leaves CALVIN on the x12 interface and comes back on the 20.0 network from a different location. So is their indeed some route-to work I need to perform here? I'm sure I'm not describing this completely, glad to provide more information. A diagram of my redundant network: http://webpages.charter.net/cycler/network.txt # CALVIN calvin# cat /etc/hostname.carp0 inet 192.168.30.1 255.255.255.0 192.168.30.255 vhid 1 advskew 1 pass foo calvin# cat /etc/hostname.carp3 inet 192.168.20.1 255.255.255.0 192.168.20.255 vhid 2 advskew 100 pass bar # HOBBES hobbes# cat /etc/hostname.carp0 inet 192.168.30.1 255.255.255.0 192.168.30.255 vhid 1 advskew 100 pass foo hobbes# cat /etc/hostname.carp3 inet 192.168.20.1 255.255.255.0 192.168.20.255 vhid 2 advskew 1 pass bar ** Both routers have an identical pf.conf. Ony the top 4 macros are different. calvin# cat /etc/pf.conf ##################################################### # Macros ##################################################### ext_if = "xl0" heartbeat_if = "dc0" server_if = "xl1" workstation_if = "xl2" web_proxy = "192.168.30.x" ssl_server = "192.168.30.x" ftp_server = "192.168.30.x" mail_server = "192.168.30.x" foo = "192.168.20.x" bar = "192.168.20.x" allow_ssh = "{xxx.xxx.xxx.xxx/32}" internal = "{ 192.168.20.0/24, 192.168.30.0/24, 10.0.0.0/24 }" ##################################################### #Options ##################################################### set loginterface $ext_if ##################################################### # NAT ##################################################### nat on $ext_if from $server_if:network to any -> ($ext_if) nat on $ext_if from $workstation_if:network to any -> ($ext_if) ##################################################### # Redirection ##################################################### # FTP rdr on $ext_if proto tcp from any to any port 21 -> $ftp_server port 21 rdr on $ext_if proto tcp from any to any port 49152:65535 -> $ftp_server port 49152:65535 #Web rdr on $ext_if proto tcp from any to any port 80 -> $web_proxy port 80 rdr on $ext_if proto tcp from any to any port 443 -> $ssl_server port 443 #Mail Traffic rdr on $ext_if proto tcp from any to any port 25 -> $mail_server port 25 rdr on $ext_if proto tcp from any to any port 110 -> $mail_server port 110 ##################################################### # setup a default deny policy ##################################################### block log all ##################################################### # pass traffic on the loopback interface in either direction ##################################################### pass quick on lo0 all ##################################################### # CARP / pfsync ##################################################### pass log on $server_if proto carp keep state pass log on $workstation_if proto carp keep state pass log on $heartbeat_if proto carp keep state pass log on $heartbeat_if proto pfsync keep state ##################################################### # Heartbeat Traffic ##################################################### pass in quick on $heartbeat_if proto { tcp, icmp } from any to any keep state ##################################################### # pass all traffic to and from the local network ##################################################### pass in quick on $server_if from $internal to any keep state flags S/SA pass out quick on $server_if from any to $internal keep state flags S/SA pass in quick on $workstation_if from $internal to any keep state flags S/SA pass out quick on $workstation_if from any to $internal keep state flags S/SA ##################################################### # Outbound ##################################################### pass out log quick on $ext_if all keep state ##################################################### # SSH Traffic ##################################################### pass in log quick on $ext_if proto tcp from $allow_ssh to any port ssh keep state flags S/SA ##################################################### # Web Traffic ##################################################### pass in log quick on $ext_if proto tcp from any to any port 80 flags S/SA keep state pass in log quick on $ext_if proto tcp from any to any port 443 flags S/SA keep state ##################################################### # Mail Traffic ##################################################### pass in log quick on $ext_if proto tcp from any to any port 25 keep state flags S/SA pass in log quick on $ext_if proto tcp from any to any port 110 keep state flags S/SA ##################################################### # Passport VPN ##################################################### pass in quick on $ext_if proto gre from xxx.xxx.xxx.xxx/32 to any keep state ##################################################### # Stupid FTP Traffic ##################################################### # in on $ext_if pass in quick on $ext_if proto tcp from any to $ftp_server port 21 keep state flags S/SA pass in quick on $ext_if proto tcp from any to $ftp_server port > 49151 keep state flags S/SA # out on $server_if pass out quick on $server_if proto tcp from any to $ftp_server port 21 keep state flags S/SA pass out quick on $server_if proto tcp from any to $ftp_server port > 49151 keep state flags S/SA # out on $workstation_if pass out quick on $workstation_if proto tcp from any to $ftp_server port 21 keep state flags S/SA pass out quick on $workstation_if proto tcp from any to $ftp_server port > 49151 keep state flags S/SA
