On Fri, 15 Nov 2013 11:31:14 -0600, Adam Thompson <[email protected]>
wrote:
> On 13-11-15 11:26 AM, Andy wrote:
>> You sir have just made my weekend! :)
>>
>> I thought that nexthop directive was a PF rule.. D'oh.. Clearly a long
>> week ;)
>>
>>> What you *might* have to do is use ifstated(8) to ensure that the
>>> "LAN" carp(4) interface always stays in sync with the "WAN" carp(4)
>>> interface. (i.e. router #1 being master for inside-facing while #2
>>> is master for outside-facing will break pf(4).)
>>
>> Absolutely.. I always put my carp interfaces into the same carp group
>> to ensure this.
>
> Now it's my turn: "into the same carp group to ensure this" - what the
> heck?
> There's nothing in carp(4) that enlightens me, and ifconfig(8) only
> talks about groups in terms of grouping pf(4) rules.
> How does that ensure that two carp(4) interfaces transition to master
> together when they're based on different physical interfaces?
> What have I missed? (Or is this yet another breakdown in OpenBSD's
> documentation?)
Hi Adam,
When you run 'ifconfig' you'll notice a 'groups:' line.
By default all carp(4) interfaces are a member of the group named 'carp',
and the pfsync(4) interface is a member of the 'pfsync' group, /and/ the
'carp' group.
This means that if the underlying physical interface which the carp(4)
interface is bound to (carpdev:) goes down, the carpdemotion counter is
incremented (by more than 1 I think) for the group 'carp' and /all/ carp(4)
interfaces belonging to the 'carp' group failover if the resulting counter
is higher than another firewalls.
You can see the current value for the carpdemotion counter for each of the
groups by running 'ifconfig -g carp' and 'ifconfig -g pfsync' etc..
In our case this all just worked by default as all carp(4) interfaces are
a member of the same group by default so it sounds like something else is
going wrong for you if you are finding individual interfaces are falling
over..
Thinking about it I have had the situation myself once or twice when one
interface fails over and not the others, but this has always been because
the CARP heartbeat protocol messages have not been passing properly between
each of the physical interface pairs from one firewall to the other etc.
I use these rules;
# carp and pfsync
if_pfsync_dev="em3"
all_carpv4_ips="<fw0.em0.physip>, <fw1.em0.physip>, <carp0.carpip>,
<fw0.em1.physip>, <fw1.em1.physip>, <em1.carpip>, <fw0.em2.physip>,
<fw1.em2.physip>, <em2.carpip>"
pass out quick proto carp keep state (no-sync) set prio 7
pass quick proto carp from { fe80::/10 } to { ff00::/8 } keep state
(no-sync)
pass quick proto carp from { $all_carpv4_ips } keep state (no-sync)
pass quick on { $if_pfsync_dev } proto pfsync keep state (no-sync)
block drop quick proto carp
NB; Replace <fw0.em0.physip> with the IP address on fw0's em0 etc.
I also connect our firewalls together via a dedicated pfsync(4) interface
with a crossover cable instead of going through a switch to improve state
sync latency.
However when rebooting a firewall, the other firewall will naturally see
its pfsync(4) interface go down as the box reboots as they are directly
connected.
This of course causes the carpdemotion counter to be incremented on both
the pfsync and carp groups which can cause problems. Especially if a
carp-interlock condition occurs (see other misc emails). So I initialise
the master with a carpdemotion value of 2 when booting up, and the backup
with a carpdemotion value of 4 when booting up to stabilise reboots (I set
the master to 2 so I can manually preempt it by decrementing the
carpdemotion on the backup to 0 when I want to do maintenance etc).
If the carpdemotion counter is the same on both firewalls the firewall
with the lower advbase decides which is master, but I have found this isn't
very reliable and can be a bit twitchy.. Hence I set the demotion counters
to set the desired master in addition to the advskew.
Hope this helps,
Andy.
