I have decided to switch my linux routers over to openbsd and as such
need to have pf up and running on them. I have a test network that I
am testing this on and am having some issues getting things working as
expected.. My network configuration is as follows:
my ascii art sux so i'll try to describe the network and provide
config files:
I have a fresh openbsd 4.2 set up with 5 physical interfaces. fxp0-3
and rl0. and carp set up on the fxp interfaces and rl0 is my pfsync
interface. carp3 is my lan interface and fxp0/carp0 is my wan
interface and default gw.
/etc/mygate: 192.168.3.158
# netstat -rn | more
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu
Interface
default 192.168.3.158 UGS 7 3923 -
carp0
10/8 link#6 UC 0 0 - rl0
10.125.221/24 link#2 UC 0 0 -
fxp0
10.126.221/24 link#3 UC 0 0 -
fxp1
10.127.221/24 link#4 UC 0 0 -
fxp2
127/8 127.0.0.1 UGRS 0 0 33208 lo0
127.0.0.1 127.0.0.1 UH 2 77 33208 lo0
172.16.10/24 link#12 UC 1 0 -
carp3
172.16.10.26 00:08:02:0b:63:59 UHLc 0 2436 -
carp3
192.168.3.128/27 link#9 UC 1 0 -
carp0
192.168.3.158 00:40:f4:76:43:62 UHLc 1 1423 -
carp0
192.168.23/24 link#5 UC 0 0 -
fxp3
192.168.45/24 link#11 UC 0 0 -
carp2
192.168.55.0/27 link#11 UC 0 0 -
carp2
224/4 127.0.0.1 URS 0 0 33208 lo0
# ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
san0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500
media: TDM t1
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0e:0c:74:6d:61
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.125.221.2 netmask 0xffffff00 broadcast 10.125.221.255
inet6 fe80::20e:cff:fe74:6d61%fxp0 prefixlen 64 scopeid 0x2
fxp1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0e:0c:3b:3f:2e
media: Ethernet autoselect (none)
status: no carrier
inet 10.126.221.2 netmask 0xffffff00 broadcast 10.126.221.255
inet6 fe80::20e:cff:fe3b:3f2e%fxp1 prefixlen 64 scopeid 0x3
fxp2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:0e:0c:74:6d:a2
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.127.221.2 netmask 0xffffff00 broadcast 10.127.221.255
inet6 fe80::20e:cff:fe74:6da2%fxp2 prefixlen 64 scopeid 0x4
fxp3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:03:47:b1:2c:c4
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.23.2 netmask 0xffffff00 broadcast 192.168.23.255
inet6 fe80::203:47ff:feb1:2cc4%fxp3 prefixlen 64 scopeid 0x5
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:50:bf:72:51:c9
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 10.23.183.1 netmask 0xff000000 broadcast 10.255.255.255
inet6 fe80::250:bfff:fe72:51c9%rl0 prefixlen 64 scopeid 0x6
enc0: flags=0<> mtu 1536
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33208
groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
carp: MASTER carpdev fxp0 vhid 1 advbase 1 advskew 0
groups: carp egress
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x9
inet 192.168.3.150 netmask 0xffffffe0 broadcast 192.168.3.159
carp1: flags=8803<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:02
carp: INIT carpdev fxp1 vhid 2 advbase 1 advskew 0
groups: carp
inet6 fe80::200:5eff:fe00:102%carp1 prefixlen 64 scopeid 0xa
inet 10.126.221.4 netmask 0xffffff00 broadcast 10.126.221.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:03
carp: MASTER carpdev fxp2 vhid 3 advbase 1 advskew 0
groups: carp
inet6 fe80::200:5eff:fe00:103%carp2 prefixlen 64 scopeid 0xb
inet 192.168.45.1 netmask 0xffffff00 broadcast 192.168.45.255
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:04
carp: MASTER carpdev fxp3 vhid 4 advbase 1 advskew 0
groups: carp
inet6 fe80::200:5eff:fe00:104%carp3 prefixlen 64 scopeid 0xc
inet 172.16.10.1 netmask 0xffffff00 broadcast 172.16.10.255
fxp0: inet 10.125.221.2 255.255.255.0 NONE
fxp1: inet 10.126.221.2 255.255.255.0 NONE
fxp2: inet 10.127.221.2 255.255.255.0 NONE
fxp3: inet 192.168.23.2 255.255.255.0 NONE
rl0: inet 10.23.183.1 255.0.0.0 NONE
carp0: inet 192.168.3.150 255.255.255.224 192.168.3.159 vhid 1
carpdev fxp0 pass tester
carp1: inet 10.126.221.4 255.255.255.0 10.126.221.255 vhid 2 carpdev
fxp1 pass tester
carp2: inet 192.168.45.1 255.255.255.0 192.168.45.255 vhid 3 carpdev
fxp2 pass tester
inet alias 192.168.55.1 255.255.255.224 192.168.55.255
carp3: inet 172.16.10.1 255.255.255.0 172.16.10.255 vhid 4 carpdev
fxp3 pass tester
pfsync0: up syncdev rl0
my pf.conf:
lan_net = "172.16.10.0/24"
set skip on lo
#set state-policy if-bound
scrub in
nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"
rdr log on fxp0 inet proto { tcp udp } from 192.168.3.96/27 to carp0
port 5900:5905 -> 172.16.10.26
rdr on fxp3 proto tcp from $lan_net to any port 21 -> 127.0.0.1 port 8021
nat log on fxp0 from $lan_net to any -> carp0
pass in on fxp0
pass out on fxp3
block in log on fxp3
pass out on fxp0 from $lan_net to any
pass in on fxp3 inet proto tcp from $lan_net port { ssh www ntp https
smtp imap imaps domain } to any
#pass in on fxp3 inet proto tcp from $lan_net port { ssh www ntp https
smtp imap imaps domain } to any no state
pass in on fxp3 inet proto udp from $lan_net port { domain ntp } to any
pass in on fxp3 inet proto icmp from $lan_net to any
# pfctl -sn
nat-anchor "ftp-proxy/*" all
nat log on fxp0 inet from 172.16.10.0/24 to any -> 192.168.3.150
rdr-anchor "ftp-proxy/*" all
rdr log on fxp0 inet proto tcp from 192.168.3.96/27 to 192.168.3.150
port 5900:5905 -> 172.16.10.26
rdr log on fxp0 inet proto udp from 192.168.3.96/27 to 192.168.3.150
port 5900:5905 -> 172.16.10.26
rdr on fxp3 inet proto tcp from 172.16.10.0/24 to any port = ftp ->
127.0.0.1 port 8021
# pfctl -sr
scrub in all fragment reassemble
pass in on fxp0 all flags S/SA keep state
pass out on fxp3 all flags S/SA keep state
block drop in log on fxp3 all
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = ssh to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = www to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = ntp to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = https to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = smtp to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = imap to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = imaps to any
flags S/SA keep state
pass in on fxp3 inet proto tcp from 172.16.10.0/24 port = domain to
any flags S/SA keep state
pass in on fxp3 inet proto udp from 172.16.10.0/24 port = domain to
any keep state
pass in on fxp3 inet proto udp from 172.16.10.0/24 port = ntp to any
keep state
pass in on fxp3 inet proto icmp from 172.16.10.0/24 to any keep state
pass out on fxp0 inet from 172.16.10.0/24 to any flags S/SA keep state
I have read the pf.faq as well as quite a few other documents about pf
and am still not figuring out what I have configured wrong. from the
carp3 network (the lan network)(and the first one i've tested so far)
I can't seem to get out to anywhere, with tcp or udp when keep state
is enabled.
When i try to telnet out on port 80, from a lan machine (telnet
www.google.com 80 and lynx www.google.com) i see the following when
watching pflog0 with tcpdump on the router/firewall:
Dec 01 12:32:32.858946 rule 2/(match) [uid 0, pid 25969] block in on
fxp3: 172.16.10.26.19360 > 216.250.190.144.53:[|domain] (ttl 64, id
46721, len 60)
Dec 01 12:32:37.889694 rule 2/(match) [uid 0, pid 25969] block in on
fxp3: 172.16.10.26.46882 > 216.250.190.145.53:[|domain] (ttl 64, id
52038, len 60)
so my dns queries are blocked even though i thought i was explicitly
passing it (along with out on fxp3. I tried it also using carp3(same
result), but from the manual you use the physical interface unless the
traffic is directed directly to the carp IP. Then i do a host lookup
for google on a different machine and put in the IP directly:
from the client machine: telnet 64.233.167.99 80
# tcpdump -n -v -ttt -e -i pflog0
tcpdump: listening on pflog0, link-type PFLOG
Dec 01 12:37:33.602184 rule 2/(match) [uid 0, pid 25969] block in on
fxp3: 172.16.10.26.11455 > 64.233.167.99.80: [|tcp] (DF) [tos 0x10]
(ttl 64, id 51850, len 64)
Dec 01 12:37:39.623354 rule 2/(match) [uid 0, pid 25969] block in on
fxp3: 172.16.10.26.11455 > 64.233.167.99.80: [|tcp] (DF) [tos 0x10]
(ttl 64, id 54981, len 64)
I did a little more reading and thought it might have to do with
states binding to particular interfaces so i tried with state-policy
set to if-bound in my pf.conf (commented out below) but same problems.
Finally I tried it with "no state" (commented out below in my pf.conf)
and got the same results:
Dec 01 12:59:48.099650 rule 2/(match) [uid 0, pid 21802] block in on
fxp3: 172.16.10.26.36264 > 216.250.190.144.53:[|domain] (ttl 64, id
11226, len 60)
Dec 01 12:59:53.131281 rule 2/(match) [uid 0, pid 21802] block in on
fxp3: 172.16.10.26.14600 > 216.250.190.145.53:[|domain] (ttl 64, id
1381, len 60)
and with ip address:
Dec 01 13:01:19.264726 rule 2/(match) [uid 0, pid 21802] block in on
fxp3: 172.16.10.26.7367 > 64.233.167.99.80: [|tcp] (DF) [tos 0x10]
(ttl 64, id 15852, len 64)
After each config change and reload of pf i flushed the states with
"pfctl -Fs" to make sure there wasn't anything sticking around.
I am sure this is some configuration error right in front of my face,
but for the life of me i'm not seeing it. Any help would be appreciated.
Thanks in advance,
Aaron
