On 12/03/12 16:21, Camiel Dobbelaar wrote:
Firewalls use dedicated interface for pfsync ($sync_if).
Are they connected directly via a cable or is there a switch in between?
Yes they have a direct cable. No switch.
I usually have "set skip" on the sync_if, if it's dedicated.
No reason why not to, I've added skip on $sync_if
Mar 9 15:46:42 firewall-2 /bsd: carp3: state transition: BACKUP -> MASTER
Mar 9 15:46:42 firewall-2 /bsd: arp_rtrequest: bad gateway value
Any idea what causes the arp_rtrequest errors? Are all your IP
addresses and netmasks sane?
According to Henning this is normal and I should ignore it. All carp
devices yes /32 netmask
see http://marc.info/?t=132187304500001&r=1&w=1 about this.
While heavily demoted, it still assumes the master role. I guess it's
not seeing the carp announcements from firewall-2 at all. Do you use
spanning tree in the network?
Yes. The latest change which I did on the switch where the firewalls are
connected is adding:
spanning-tree portfast trunk
spanning-tree bpdufilter enable
in order to startup the port faster. Don't know if this is causing the
problem, cause now the ports are coming up really fast. They used to
come up after 1 minute.
I will check without these 2 options as this is how it was so far.
Ports (external/internal) looks like this now:
interface GigabitEthernet1/24
description firewall-1-ext
no ip address
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx
switchport mode trunk
spanning-tree portfast trunk
spanning-tree bpdufilter enable
end
#show spanning-tree interface gigabitEthernet 1/24
Vlan Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- --------
--------------------------------
VLANXXX Desg FWD 4 128.24 Edge P2p
Mar 9 15:49:12 firewall-1 /bsd: carp1: state transition: BACKUP -> MASTER
Mar 9 15:49:12 firewall-1 /bsd: arp_rtrequest: bad gateway value
Mar 9 15:49:12 firewall-1 /bsd: carp0: state transition: BACKUP -> MASTER
Mar 9 15:49:12 firewall-1 /bsd: arp_rtrequest: bad gateway value
Mar 9 15:49:13 firewall-1 /bsd: carp3: state transition: BACKUP -> MASTER
Mar 9 15:49:13 firewall-1 /bsd: arp_rtrequest: bad gateway value
Mar 9 15:49:13 firewall-1 /bsd: carp2: state transition: BACKUP -> MASTER
Mar 9 15:49:13 firewall-1 /bsd: arp_rtrequest: bad gateway value
Manually enforce BACKUP mode
firewall-1# ifconfig -g carp carpdemote
Here it gets weird... it's already at demote=128, so adding one more
shouldn't help. I suspect it would have gone to backup anyway.
Well it didn't. I ssh to machine and then issued the command. As you can
see it took me
18 seconds, so if it could automatically change to backup it would have
already.
Mar 9 15:49:31 firewall-1 /bsd: carp1: state transition: MASTER -> BACKUP
Mar 9 15:49:31 firewall-1 /bsd: carp0: state transition: MASTER -> BACKUP
Mar 9 15:49:31 firewall-1 /bsd: carp2: state transition: MASTER -> BACKUP
Mar 9 15:49:31 firewall-1 /bsd: carp3: state transition: MASTER -> BACKUP
This is around 30 seconds after the first boot message... sounds like
the switch again that blocks the traffic on the port for 30 seconds.
How many states do you typically have? The bulk pfsync is taking a
really long time here... 4 minutes. Any errors on the pfsync interface?
What speed is it?
I usually have around 90k states (pfctl -ss |wc -l)
On both firewalls it's 1Gbps
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
# netstat -id
Name Mtu Network Address Ipkts Ierrs Opkts
Oerrs Colls Drop
em2(sync_if_f1) 1500<Link> 00:19:99:98:e4:ea 682406 225 255969304
0 0 0
bge1(sync_if_f2) 1500<Link> 00:0a:e4:80:73:3d 387753797 461 1152887
0 0 0
f1# netstat -s
carp:
12 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
1586084 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
8 transitions to master
pfsync:
682381 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
88 stale states
809627 failed state lookup/inserts
256080550 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
f2# netstat -s
carp:
2236176 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
460 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
12 transitions to master
pfsync:
387828563 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
435 stale states
1173653 failed state lookup/inserts
1152819 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
What does your ifstated.conf look like?
ifstated runs only on primary firewall.
Primary firewall runs with advbase 1 advskew 10
secondary firewall runs with advbase 1 advskew 100
carp_up = "carp0.link.up&& carp1.link.up&& carp2.link.up&& carp3.link.up"
carp_down = "!carp0.link.up&& !carp1.link.up&& !carp2.link.up&&
!carp3.link.up"
carp_sync = "carp0.link.up&& carp1.link.up&& carp2.link.up&& carp3.link.up
|| \
!carp0.link.up&& !carp1.link.up&& !carp2.link.up&& !carp3.link.up"
# check remote gateways
net = '( "ping -q -c 1 -w 1 aaa.aaa.aaa.aaa> /dev/null" every 10&& \
"ping -q -c 1 -w 1 bbb.bbb.bbb.bbb> /dev/null" every 10&& \
"ping -q -c 1 -w 1 ccc.ccc.ccc.ccc> /dev/null" every 10&& \
"ping -q -c 1 -w 1 ddd.ddd.ddd.ddd> /dev/null" every 10)'
# check firewall-2
peer = '( "ping -q -c 1 -w 1 eee.eee.eee.eee> /dev/null" every 10 )'
state auto {
if $carp_up
set-state primary
if $carp_down
set-state backup
}
state primary {
init {
run "ifconfig carp0 advskew 10"
run "ifconfig carp1 advskew 10"
run "ifconfig carp2 advskew 10"
run "ifconfig carp3 advskew 10"
}
if ! $net
set-state demoted
}
state demoted {
init {
run "ifconfig carp0 advskew 200"
run "ifconfig carp1 advskew 200"
run "ifconfig carp2 advskew 200"
run "ifconfig carp3 advskew 200"
}
if $net
set-state primary
}
state promoted {
init {
run "ifconfig carp0 advskew 101"
run "ifconfig carp1 advskew 101"
run "ifconfig carp2 advskew 101"
run "ifconfig carp3 advskew 101"
}
if $net
set-state primary
if ! $net&& $peer
set-state backup
}
state backup {
init {
run "ifconfig carp0 advskew 254"
run "ifconfig carp1 advskew 254"
run "ifconfig carp2 advskew 254"
run "ifconfig carp3 advskew 254"
}
# The "sleep 5" below is a hack to dampen the $carp_sync when we come
# out of promoted state. Thinking about the correct fix...
if ! $carp_sync&& $net&& "sleep 5" every 10
if ! $carp_sync&& $net
set-state promoted
}
regards,
Giannis
