I have got a weird problem with my network setup.
I have a pair of identical OpenBSD 4.3 (stable, GENERIC) boxes running in
Active/Standby failover using carp, pfsync and sasyncd
uname: OpenBSD nitehawk.contoso.com 4.3 GENERIC#698 i386
The CARP boxes external interface (bge0) are: 172.21.171.{6, 7} and they share
172.21.171.5
The CARP boxes internal interface (bge1) are: 172.21.100.{2, 3} and they share
172.21.100.1
They failover interface (em0) are: 172.21.123.{2,3}
I have a pair of Cisco ASA (Active/Standby failover) behind the CARP
boxes on the bge1 interface. The ASA interface IP is 172.21.100.4
The ASA has one network behind it (172.21.69.0/24)
There are 5 routers apart from the CARP boxes on the bge0 interface. I have
two separate IPsec tunnels terminating on two Cisco 2811 routers. The
cisco routers have one network behind them, 192.168.171/24 and
192.168.101/24 respectively
One of my requirement is to do policy based static NAT with
172.21.69.0/24. To explain it more, when the traffic is to/from
192.168.101.0/24, the ASA would static NAT 172.21.69.0/24 to
172.21.169.0/24
The interesting traffic for the second tunnel essentially is:
192.168.101.0/24 <=> 172.21.169.0/24. For completeness, the interesting
traffic for the first tunnel is 192.168.171.0/24 <=> 172.21.69.0/24
netstat -rnf encap on the master:
nitehawk (OpenBSD): [~]
ttyp0: [700]# netstat -rnf encap
Routing tables
Encap:
Source Port Destination Port Proto
SA(Address/Proto/Type/Direction)
192.168.101/24 0 172.21.169/24 0 0 172.21.171.9/esp/use/in
172.21.169/24 0 192.168.101/24 0 0 172.21.171.9/esp/require/out
192.168.171/24 0 172.21.69/24 0 0 172.21.171.8/esp/use/in
172.21.69/24 0 192.168.171/24 0 0 172.21.171.8/esp/require/out
I have enabled OSPF routing on all network devices (i.e. CARP boxes, 5
routers on the outside and the ASA on the inside)
ifconfig on the master:
nitehawk (OpenBSD): [~]
ttyp0: [686]# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:15:17:51:81:75
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 172.21.123.2 netmask 0xffffff00 broadcast 172.21.123.255
inet6 fe80::215:17ff:fe51:8175%em0 prefixlen 64 scopeid 0x1
inet6 fd1b:d92f:84f3:123:215:17ff:fe51:8175 prefixlen 64
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:1c:23:e1:cb:85
groups: egress
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 172.21.171.6 netmask 0xffffff00 broadcast 172.21.171.255
inet6 fe80::21c:23ff:fee1:cb85%bge0 prefixlen 64 scopeid 0x2
inet6 fd1b:d92f:84f3:171:21c:23ff:fee1:cb85 prefixlen 64
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:1c:23:e1:cb:86
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 172.21.100.2 netmask 0xffffff00 broadcast 172.21.100.255
inet6 fe80::21c:23ff:fee1:cb86%bge1 prefixlen 64 scopeid 0x3
inet6 fd1b:d92f:84f3:100:21c:23ff:fee1:cb86 prefixlen 64
enc0: flags=0<> mtu 1536
lo127: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33208
groups: lo
inet 172.21.127.6 netmask 0xffffffff
inet6 fd1b:d92f:84f3:127:31e1:bb3f:20c8:7f06 prefixlen 128
pfsync0: flags=41<UP,RUNNING> mtu 1460
pfsync: syncdev: em0 syncpeer: 224.0.0.240 maxupd: 128
groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33208
groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:01
carp: MASTER carpdev bge0 vhid 1 advbase 1 advskew 0
groups: carp
inet 172.21.171.5 netmask 0xffffff00 broadcast 172.21.171.255
inet6 fe80::200:5eff:fe00:101%carp0 prefixlen 64 scopeid 0x7
inet6 fd1b:d92f:84f3:171:f9a0:3201:525c:671 prefixlen 64
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:00:5e:00:01:02
carp: MASTER carpdev bge1 vhid 2 advbase 1 advskew 0
groups: carp
inet 172.21.100.1 netmask 0xffffff00 broadcast 172.21.100.255
inet6 fe80::200:5eff:fe00:102%carp1 prefixlen 64 scopeid 0x8
inet6 fd1b:d92f:84f3:100:1983:b905:3d8e:3dc7 prefixlen 64
When 172.21.69.17 (behind the ASA) tries to talk to 192.168.171.0/24, I
can see that the traffic is reaching the bge1 interface, then getting
encapsulated in ESP tunnel and then sent across. In short, works as
expected.
Example:
From: 172.21.69.17 (behind ASA)
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
groups: lo
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:19:b9:f7:5f:2d
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 172.21.69.17 netmask 0xffffff00 broadcast 172.21.69.255
inet6 fe80::219:b9ff:fef7:5f2d%bge0 prefixlen 64 scopeid 0x1
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:19:b9:f7:5f:2e
media: Ethernet autoselect (1000baseT full-duplex)
status: active
inet 192.168.153.1 netmask 0xffffff00 broadcast 192.168.153.255
inet6 fe80::219:b9ff:fef7:5f2e%bge1 prefixlen 64 scopeid 0x2
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33224
pfsync0: flags=0<> mtu 1460
groups: carp
enc0: flags=0<> mtu 1536
# ping -c 1 192.168.171.17
PING 192.168.171.17 (192.168.171.17): 56 data bytes
64 bytes from 192.168.171.17: icmp_seq=0 ttl=253 time=1.632 ms
--- 192.168.171.17 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.632/1.632/1.632/0.000 ms
Capture on bge0 (on the master):
nitehawk (OpenBSD): [~]
ttyp2: [688]# tcpdump -envs 1500 -i bge0 proto 50
tcpdump: listening on bge0, link-type EN10MB
11:47:26.621234 00:1c:23:e1:cb:85 00:1f:ca:9c:b0:30 0800 166: esp 172.21.171.5 >
172.21.171.8 spi 0xD8B34605 seq 6 len 132 (ttl 64, id 60796, len 152)
11:47:26.622262 00:1f:ca:9c:b0:30 00:00:5e:00:01:01 0800 166: esp 172.21.171.8 >
172.21.171.5 spi 0x92557359 seq 6 len 132 (ttl 255, id 9054, len 152)
Capture on bge1 (on the master):
nitehawk (OpenBSD): [~]
ttyp0: [689]# tcpdump -envs 1500 -i bge1 icmp
tcpdump: listening on bge1, link-type EN10MB
11:47:26.621145 00:1f:9e:50:8d:5c 00:1c:23:e1:cb:86 0800 98: 172.21.69.17 >
192.168.171.17: icmp: echo request (id:ff53 seq:0) (ttl 255, id 3547, len 84)
11:47:26.622316 00:1c:23:e1:cb:86 00:1f:9e:50:8d:5c 0800 98: 192.168.171.17 >
172.21.69.17: icmp: echo reply (id:ff53 seq:0) (ttl 253, id 43508, len 84)
However, if I ping 192.168.101.17, I see that the packet gets across,
but the return packet never makes it to the bge1 interface, not even on the enc0
interface:
On 172.21.69.17:
# ping -c 1 192.168.101.17
PING 192.168.101.17 (192.168.101.17): 56 data bytes
--- 192.168.101.17 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
Capture on bge0 (on the master)
nitehawk (OpenBSD): [~]
ttyp1: [705]# tcpdump -envs 1500 -i bge0 proto 50
tcpdump: listening on bge0, link-type EN10MB
12:15:52.625742 00:1c:23:e1:cb:85 00:1f:9e:56:83:f0 0800 166: esp 172.21.171.5 >
172.21.171.9 spi 0x7F3CFA74 seq 6 len 132 (ttl 64, id 2500, len 152)
12:15:52.626772 00:1f:9e:56:83:f0 00:00:5e:00:01:01 0800 166: esp 172.21.171.9 >
172.21.171.5 spi 0x95103468 seq 6 len 132 (ttl 255, id 4623, len 152)
Capture on bge1 (on the master)
nitehawk (OpenBSD): [~]
ttyp2: [706]# tcpdump -envs 1500 -i bge1 icmp
tcpdump: listening on bge1, link-type EN10MB
12:15:52.625654 00:1f:9e:50:8d:5c 00:1c:23:e1:cb:86 0800 98: 172.21.169.17 >
192.168.101.17: icmp: echo request (id:6346 seq:0) (ttl 255, id 10217, len 84)
nitehawk (OpenBSD): [~]
ttyp0: [706]# tcpdump -env -i enc0
tcpdump: listening on enc0, link-type ENC
12:15:52.625691 (authentic,confidential): SPI 0x7f3cfa74: 172.21.171.5 >
172.21.171.9: 172.21.169.17 > 192.168.101.17: icmp: echo request (id:6346 seq:0)
(ttl 254, id 10217, len 84) (ttl 64, id 2500, len 104, bad cksum 0!)
route -n get and netstat -rnf inet and ospfctl show rib :
nitehawk (OpenBSD): [~]
ttyp0: [696]# route -n get 172.21.169.17
route to: 172.21.169.17
destination: 172.21.169.0
mask: 255.255.255.0
gateway: 172.21.100.4
interface: bge1
if address: 172.21.100.2
flags: <UP,GATEWAY,DONE,PROTO2>
use hopcount mtu expire
0 0 0 0
nitehawk (OpenBSD): [~]
ttyp0: [697]# netstat -rnf inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Interface
default 172.21.171.1 UG2 0 0 - bge0
127/8 127.0.0.1 UGRS 0 0 33208 lo0
127.0.0.1 127.0.0.1 UH 1 240 33208 lo0
172.21.50/24 172.21.171.4 UG2 0 0 - bge0
172.21.52/24 172.21.171.1 UG2 0 0 - bge0
172.21.69/24 172.21.100.4 UG2 0 1689 - bge1
172.21.99/24 172.21.171.4 UG2 0 0 - bge0
172.21.100/24 link#3 UC 2 0 - bge1
172.21.100.1 172.21.100.1 UH 0 0 - carp1
172.21.100.3 00:1c:23:e1:9c:b5 UHLc 1 54723 - bge1
172.21.100.4 00:1f:9e:50:8d:5c UHLc 2 2828 - bge1
172.21.123/24 link#1 UC 1 0 - em0
172.21.123.3 00:15:17:51:81:bd UHLc 2 462015 - em0
172.21.125/24 172.21.171.2 UG2 1 5806 - bge0
172.21.127.1/32 172.21.171.1 UG2 0 0 - bge0
172.21.127.4/32 172.21.171.4 UG2 0 0 - bge0
172.21.127.6 172.21.127.6 UH 0 0 33208 lo127
172.21.127.7/32 172.21.123.3 UG2 0 0 - em0
172.21.127.7/32 172.21.100.3 UG2 0 0 - bge1
172.21.127.7/32 172.21.171.7 UG2 0 0 - bge0
172.21.127.8/32 172.21.171.8 UG2 0 0 - bge0
172.21.127.9/32 172.21.171.9 UG2 0 0 - bge0
172.21.145/24 172.21.171.2 UG2 0 0 - bge0
172.21.167/24 172.21.171.2 UG2 3 1136 - bge0
172.21.169/24 172.21.100.4 UG2 0 0 - bge1
172.21.171/24 link#2 UC 6 0 - bge0
172.21.171.1 00:0e:83:f6:0e:5d UHLc 3 51829 - bge0
172.21.171.2 00:15:c5:2a:e6:9d UHLc 3 1 - bge0
172.21.171.4 link#2 UHLc 3 1 - bge0
172.21.171.5 172.21.171.5 UH 0 0 - carp0
172.21.171.7 00:1c:23:e1:9c:b4 UHLc 1 51810 - bge0
172.21.171.8 00:1f:ca:9c:b0:30 UHLc 3 2214 - bge0
172.21.171.9 00:1f:9e:56:83:f0 UHLc 2 3065 - bge0
192.168.101/24 172.21.171.9 UG2 0 0 - bge0
192.168.171/24 172.21.171.8 UG2 0 0 - bge0
224/4 127.0.0.1 URS 0 0 33208 lo0
nitehawk (OpenBSD): [~]
ttyp0: [698]# ospfct show rib
ksh: ospfct: not found
nitehawk (OpenBSD): [~]
ttyp0: [699]# ospfctl show rib
Destination Nexthop Path Type Type Cost Uptime
172.21.127.1 172.21.171.1 Intra-Area Router 10 3d20h02m
172.21.127.2 172.21.171.2 Intra-Area Router 10 3d20h02m
172.21.127.4 172.21.171.4 Intra-Area Router 10 3d20h02m
172.21.127.7 172.21.123.3 Intra-Area Router 10 6d00h00m
172.21.127.7 172.21.171.7 Intra-Area Router 10 3d20h02m
172.21.127.7 172.21.100.3 Intra-Area Router 10 3d20h02m
172.21.127.10 172.21.100.4 Intra-Area Router 10 3d20h02m
172.21.50.0/24 172.21.171.4 Intra-Area Network 11 3d20h02m
172.21.69.0/24 172.21.100.4 Intra-Area Network 20 3d20h02m
172.21.99.0/24 172.21.171.4 Intra-Area Network 11 3d20h02m
172.21.100.0/24 172.21.100.2 Intra-Area Network 10 6d00h01m
172.21.123.0/24 172.21.123.2 Intra-Area Network 10 6d00h00m
172.21.127.1/32 172.21.171.1 Intra-Area Network 11 3d20h02m
172.21.127.4/32 172.21.171.4 Intra-Area Network 11 3d20h02m
172.21.127.7/32 172.21.123.3 Intra-Area Network 20 6d00h00m
172.21.127.7/32 172.21.171.7 Intra-Area Network 20 3d20h02m
172.21.127.7/32 172.21.100.3 Intra-Area Network 20 3d20h02m
172.21.127.8/32 172.21.171.8 Intra-Area Network 11 3d20h02m
172.21.127.9/32 172.21.171.9 Intra-Area Network 11 3d20h02m
172.21.171.0/24 172.21.171.6 Intra-Area Network 10 3d20h02m
192.168.101.0/24 172.21.171.9 Intra-Area Network 11 3d20h02m
192.168.171.0/24 172.21.171.8 Intra-Area Network 11 3d20h02m
0.0.0.0/0 172.21.171.1 Type 2 ext Network 10 3d20h02m
172.21.52.0/24 172.21.171.1 Type 2 ext Network 10 3d20h02m
172.21.125.0/24 172.21.171.2 Type 2 ext Network 10 3d20h02m
172.21.145.0/24 172.21.171.2 Type 2 ext Network 10 3d20h02m
172.21.167.0/24 172.21.171.2 Type 2 ext Network 10 3d20h02m
172.21.169.0/24 172.21.100.4 Type 2 ext Network 20 00:39:21
Can anybody explain, why the BSD box blackholes the route back, even
though it is the routing table with the proper next hop?
