Hello,
I have configured two firewalls on IBM x306 servers (they have 512MB
of ram, 3.0 GHz PIV, and 2 gigabit ethernet cards Intel PRO/1000). I
have also added one more network card: "D-Link Systems 530TX+" which
is dedicated to pfsync and connected to both firewalls with crossover
Ethernet cable. Now they are running OpenBSD 3.8, but earlier I have
setup with 3.7. Systems installations are almost default, with default
kernels.
I have configured 35 carp addresses. Thay are filtered and redirected
to internal network on firewalls. Both machines have identical pf.conf
and almost identical carp configurations. On fw2 which is backup carp
server I have only added advskew parameter.
My sysctl configuration:
net.inet.carp.log=1
net.inet.carp.preempt=1
net.inet.carp.allow=1
net.inet.carp.arpbalance=1
My interfaces configurations:
Configuration on fw1:
# ifconfig em0
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:25:ab:b9:76
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet EXTERNAL_IP netmask 0xffffff00 broadcast EXTERNAL_MASK
inet6 fe80::211:25ff:feab:b976%em0 prefixlen 64 scopeid 0x1
# ifconfig em1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:25:ab:b9:77
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::211:25ff:feab:b977%em1 prefixlen 64 scopeid 0x3
# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:95:5e:f3:03
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
inet6 fe80::211:95ff:fe5e:f303%rl0 prefixlen 64 scopeid 0x2
# ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> mtu 1348
pfsync: syncdev: rl0 syncpeer: 224.0.0.240 maxupd: 128
# ifconfig carp4
carp4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
carp: MASTER carpdev em0 vhid 5 advbase 1 advskew 0
groups: carp
inet CARP_EXTERNAL_IP netmask 0xffffff00 broadcast CARP_EXTERNAL_MASK
Sample carp interface configuration:
# cat /etc/hostname.carp4
inet CARP_EXTERNAL_IP 255.255.255.0 CARP_EXTERNAL_MASK vhid 5 pass
jk12h3jk12h3jk12h3jk2h13jkh12kj3hk12jh3
Configuration on fw2:
# ifconfig em0
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:25:57:e1:96
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet EXTERNAL_IP2 netmask 0xffffff00 broadcast EXTERNAL_MASK
inet6 fe80::211:25ff:fe57:e196%em0 prefixlen 64 scopeid 0x1
# ifconfig em1
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:25:57:e1:97
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::211:25ff:fe57:e197%em1 prefixlen 64 scopeid 0x3
# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:11:95:5e:f3:0e
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255
inet6 fe80::211:95ff:fe5e:f30e%rl0 prefixlen 64 scopeid 0x2
# ifconfig pfsync0
pfsync0: flags=41<UP,RUNNING> mtu 1348
pfsync: syncdev: rl0 syncpeer: 224.0.0.240 maxupd: 128
# ifconfig carp4
carp4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
carp: MASTER carpdev em0 vhid 5 advbase 1 advskew 10
groups: carp
inet CARP_EXTERNAL_IP netmask 0xffffff00 broadcast CARP_EXTERNAL_MASK
# cat /etc/hostname.carp4
inet CARP_EXTERNAL_IP 255.255.255.0 CARP_EXTERNAL_MASK vhid 5 advskew
10 pass jk12h3jk12h3jk12h3jk2h13jkh12kj3hk12jh3
My pf configuration:
pf.conf: (I have simplify configuration, to only one internal server).
ext_if="em0"
int_if="em1"
sync_if="rl0"
int_net="192.168.0.0/24"
fw1="EXTERNAL_IP"
fw1_int="192.168.0.1"
fw2="EXTERNAL_IP"
fw2_int="192.168.0.2"
table <ALLOW> persist { $host1, $host2 }
table <OURNETWORKS> persist { our networks definitions}
carp_ext_addr4="EXTERNAL_IP"
server_ssh="192.168.0.18"
##########
# OPTIONS
##########
set limit states 100000
set limit src-nodes 100000
set limit frags 50000
set optimization normal
#set debug loud
set block-policy drop
##########
# SCRUB
##########
scrub on $ext_if all random-id
##########
# NAT
##########
nat pass on $ext_if inet proto tcp from $server_ssh to <OURNETWORKS>
port 80 -> $carp_ext_addr4
nat pass on $ext_if inet proto tcp from $server_ssh to <OURNETWORKS>
port 443 -> $carp_ext_addr4
###########
# RDR
###########
# redirect ssh
rdr on $ext_if proto tcp from <ALLOW> to $carp_ext_addr4 port 22 ->
$server_ssh port 22
# redirect https
rdr on $ext_if proto tcp from <ALLOW> to $carp_ext_addr4 port 443 ->
$server_ssh port 443
block log(all) all
pass quick on { lo } keep state
antispoof quick for { lo }
pass quick on $sync_if proto pfsync keep state
pass quick on $ext_if proto carp keep state
pass quick on $int_if proto carp keep state
#pings
pass in inet proto icmp icmp-type echoreq code 0 keep state
#connections from firewalls
pass out on $ext_if from { $fw1, $fw2 } to any keep state
#connections to local nets
pass out on $int_if from { $fw1_int, $fw2_int } to $int_net keep state
pass in on $int_if from { $fw1_int, $fw2_int } to $int_net keep state
# connections to firewalls
pass in on $ext_if proto tcp from <ALLOW> to { $fw1, $fw2 } port ssh keep state
###########
# SERVER
###########
# redirect ssh
pass in on $ext_if proto tcp from <ALLOW> to $server_ssh port 22 keep state
pass out on $int_if proto tcp from <ALLOW> to $server_ssh port 22 keep state
# redirect https
pass in on $ext_if proto tcp from <ALLOW> to $server_ssh port 443 keep state
pass out on $int_if proto tcp from <ALLOW> to $server_ssh port 443 keep state
# connections from server
pass in on $int_if proto tcp from $server_ssh to <OURNETWORKS> port {
80, 443 } keep state
pass out on $ext_if proto tcp from $server_ssh to <OURNETWORKS> port {
80, 443 } keep state
I have some problems with this configuration:
1. On fw2 which is backup firewall I have carp interfaces (not all of
them, maybe half or more) in MASTER state. This is strange, because in
the same time on fw1 all interfaces have MASTER status.
on fw2 I have:
# ifconfig -a | grep BACKUP
carp: BACKUP carpdev em1 vhid 1 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 2 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 11 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 19 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 20 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 28 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 29 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 6 advbase 1 advskew 10
carp: BACKUP carpdev em0 vhid 7 advbase 1 advskew 10
# ifconfig -a | grep MASTER
carp: MASTER carpdev em0 vhid 12 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 13 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 14 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 15 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 16 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 17 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 18 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 3 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 21 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 22 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 23 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 24 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 25 advbase 1 advskew 10
carp: MASTER carpdev em0 vhid 26 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 27 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 30 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 4 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 31 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 32 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 33 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 34 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 35 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 5 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 8 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 9 advbase 1 advskew 100
carp: MASTER carpdev em0 vhid 10 advbase 1 advskew 100
As a result of this, when I have ping one of the carp interfaces which
is in MASTER state on both firewalls, i get duplicated replays
2. Problems with interfaces statistics:
# netstat -p carp -s
carp:
64322654 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
61597498 discarded for bad vhid
0 discarded because of a bad address list
31593122 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
# netstat -p pfsync -s
pfsync:
42614 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
33835 failed state lookup/inserts
3292926 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
13937 send error
As You can see here, I have huge count of "discarded for bad vhid"
packets, and with "failed state lookup/inserts". I think that this
may be helpful in my problem.
3. In result of "pfctl -ss" I have see:
#pfctl -ss
all tcp 192.168.0.118:80 <- SOME_CARP_IP:80 <- EXTIP:63647
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.118:443 <- SOME_CARP_IP:443 <- EXTIP:63915
ESTABLISHED:ESTABLISHED
all tcp 192.168.0.118:443 <- SOME_CARP_IP:443 <- EXTIP:64953
FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.0.119:443 <- SOME_CARP_IP:443 <- EXTIP:55619
FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.0.119:80 <- SOME_CARP_IP:80 <- EXTIP:62448
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.120:80 <- SOME_CARP_IP:80 <- EXTIP:64807
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.121:443 <- SOME_CARP_IP:443 <- EXTIP:60305
FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.0.122:443 <- SOME_CARP_IP:443 <- EXTIP:64273
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.122:80 <- SOME_CARP_IP:80 <- EXTIP:57573
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.123:443 <- SOME_CARP_IP7:443 <- EXTIP:59520
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.123:80 <- SOME_CARP_IP7:80 <- EXTIP:51947
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.124:443 <- SOME_CARP_IP6:443 <- EXTIP:62809
CLOSING:CLOSING
all tcp 192.168.0.124:443 <- SOME_CARP_IP6:443 <- EXTIP:59778
FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.0.124:80 <- SOME_CARP_IP5:80 <- EXTIP:59633
TIME_WAIT:TIME_WAIT
all tcp 192.168.0.125:80 <- SOME_CARP_IP5:80 <- EXTIP:65042
FIN_WAIT_2:FIN_WAIT_2
all tcp 192.168.0.126:443 <- SOME_CARP_IP4:443 <- EXTIP:55371
CLOSING:CLOSING
all tcp 192.168.0.126:80 <- SOME_CARP_IP4:80 <- EXTIP:51908
CLOSING:CLOSING
all udp SOME_EXT_IP:514 <- 192.168.0.15:514 NO_TRAFFIC:SINGLE
all udp SOME_EXT_IP:514 <- 192.168.0.17:514 NO_TRAFFIC:SINGLE
all udp SOME_EXT_IP:514 <- 192.168.0.19:514 NO_TRAFFIC:SINGLE
all udp 192.168.0.15:514 -> SOME_CARP_IP3:62066 -> SOME_EXT_IP:514
SINGLE:NO_TRAFFIC
all udp 192.168.0.17:514 -> SOME_CARP_IP3:55176 -> SOME_EXT_IP:514
SINGLE:NO_TRAFFIC
all udp 192.168.0.218:514 -> SOME_CARP_IP2:55103 -> SOME_EXT_IP:514
SINGLE:NO_TRAFFIC
all udp 192.168.0.216:514 -> SOME_CARP_IP1:54509 -> SOME_EXT_IP:514
SINGLE:NO_TRAFFIC
all udp 192.168.0.19:514 -> SOME_CARP_IP2:54721 -> SOME_EXT_IP:514
SINGLE:NO_TRAFFIC
all udp SOME_EXT_IP:514 <- 192.168.0.216:514 NO_TRAFFIC:SINGLE
all udp SOME_EXT_IP:514 <- 192.168.0.218:514 NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 192.168.0.1 NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 195.187.142.11 NO_TRAFFIC:SINGLE
all carp 224.0.0.18 <- 195.187.142.12 4:1
all carp 192.168.0.1 -> 224.0.0.18 SINGLE:NO_TRAFFIC
all carp 195.187.142.11 -> 224.0.0.18 SINGLE:NO_TRAFFIC
all carp 195.187.142.12 -> 224.0.0.18 SINGLE:NO_TRAFFIC
all 160 c0a8:76::[443] <- c3bb:8e25::[443] <- d999:a1e8::[55600]
NO_TRAFFIC:MULTIPLE
all 160 d999:a1e8::[58297] <- c0a8:74::[80] 0:4
all 160 f9a9:e136:0:0:200::[17312] <- ::e303:49e3:e304:4987 <-
481::[41809] NO_TRAFFIC:MULTIPLE
all 160 c0a8:71::[443] <- c3bb:8e20::[443] <- d999:a1e8::[55642]
NO_TRAFFIC:MULTIPLE
all 160 c0a8:6a::[443] <- c3bb:8e19::[443] <- d999:a1e8::[52404] 0:4
all 160 d999:a1e8::[59921] <- c0a8:7c::[443] 4:4
all 160 d999:a1e8::[53605] <- c0a8:6b::[443] NO_TRAFFIC:MULTIPLE
all 160 c0a8:7b::[443] <- c3bb:8e2a::[443] <- d999:a1e8::[52964] 0:9
all pfsync 172.16.1.1 -> 172.16.1.2 MULTIPLE:MULTIPLE
all pfsync 224.0.0.240 <- 172.16.1.2 NO_TRAFFIC:SINGLE
all pfsync 172.16.1.1 -> 224.0.0.240 SINGLE:NO_TRAFFIC
"áműm3] mux 8c06::230:90d6:fe07 <- f6d0:0:0:fe07::30[37078] <-
0:8:0:e000:12:64:9cd6:64[40150] NO_TRAFFIC:NO_TRAFFIC
Öţ mux 0:fe07::28:98d6:0 <- 0:e000:12:69:98d6:69:98d6:6903[50107] <-
39b5:3041:e41e:67ea:a7c6:42dc:fe9b:2ab1[11701] 12:0
tcp 219.210.0.0:5968 -> 0.0.0.0 -> 0.0.0.0 CLOSED:ESTABLISHED
pup f6d0:0:0:fe07::68[38358] <- 0:8:0:e000:12:b8:77d6:b8[30678] <-
6338:fa98:c029:e7b2:859e:a517:e7a8:399e[1503]
NO_TRAFFIC:NO_TRAFFIC
160 100:: <- ::[45274] <- c00::[64556] 9:4
ű!˙Ä 160 481::[10833] <- 0:0:ffff:0:481::[63913] <- 200::[61190] 4:4
{Öţ mux 0:fe07::f0:7bd6:0 <- 0:e000:12:2:9dd6:2:9dd6:3452[29154] <-
0:840b:9623:50e2:650c:4671:24d4:92a2[28094]
NO_TRAFFIC:NO_TRAFFIC
{Öţ mux 0:fe07::38:7bd6:0 <- 0:e000:12:49:8ed6:49:8ed6:48f9[38204] <-
b9c8:8724:42e:452b:b537:4651:710e:eef4[63698]
NO_TRAFFIC:NO_TRAFFIC
nż_ÖnŔd 160 481::[58123] <- 0:0:8218:0:981::[63913] <- 300::cb01
NO_TRAFFIC:NO_TRAFFIC
őşäfőťč 160 481::[14411] <- 0:0:8218:0:481::[63913] <- 200::[61851] 7:4
ůŠá6 ip 1016:f9a9:100:5e00:12:0:5e00:118[2048] <-
4000:ff70:26c2:c3bb:8e0c:e000:12:2118[2567] <-
5def:60da:aff3:c61d:c53:9ea:d92f:3976[64981] NO_TRAFFIC:SINGLE
These bottom states are quite odd. When I have disable pfsync
interfaces these states don't show up.
Is there any solution to resolve my problems with carp? If there is
necessary to show You more informations on my current configuration I
will do everything what I only can.
I will appreciate any help.
best regards,
Marcin Miksowski
