Re: Can't get carp to fail over all interfaces with pfsync
On Wed, Nov 11, 2009 at 05:47:52PM +, Stuart Henderson wrote: On 2009-11-10, Daniel Ouellet dan...@presscom.net wrote: FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass password $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass password $ cat /etc/hostname.pfsync0 Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one? It sees them as different, because they're on different interfaces. I believe I now remember why I vhids should be different The lladdr of a carp interface is 00:00:5e:00:01:vhid. Now in the (atypical, I know) case that both carp interface share the same physical network, this is asking for trouble. Also, even if they do not share the same network, it is still confusing when reading logs etc. That's why I consider it a good thing to avoid reusing vhids. -Otto
Re: Can't get carp to fail over all interfaces with pfsync
On 2009-11-10, Daniel Ouellet dan...@presscom.net wrote: FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass password $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass password $ cat /etc/hostname.pfsync0 Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one? It sees them as different, because they're on different interfaces.
Re: Can't get carp to fail over all interfaces with pfsync
On Tue, Nov 10, 2009 at 06:36:24PM +1100, Mikel Lindsaar wrote: On Tue, Nov 10, 2009 at 5:37 PM, Daniel Ouellet dan...@presscom.net wrote: FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass password $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass password $ cat /etc/hostname.pfsync0 Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one? Initially I had them running as different VHIDs. carp0 was vhid 1 and carp1 was vhid 2, however, this did not work either... plus I would get unknown vhid errors in the netstat -s -p pfsync output if I had different vhids. Mikel Then you did something else wrong, like forgetting to change them on both hosts. Different carp interfaces should have different vhids. Also, a common error is to have (slightly) different ip's, netmasks or aliases on the the carp interfaces for the two hosts. -Otto
Re: Can't get carp to fail over all interfaces with pfsync
On Tue, Nov 10, 2009 at 7:25 PM, Otto Moerbeek o...@drijf.net wrote: On Tue, Nov 10, 2009 at 06:36:24PM +1100, Mikel Lindsaar wrote: Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one? Initially I had them running as different VHIDs. carp0 was vhid 1 and carp1 was vhid 2, however, this did not work either... plus I would get unknown vhid errors in the netstat -s -p pfsync output if I had different vhids. Mikel Then you did something else wrong, like forgetting to change them on both hosts. Different carp interfaces should have different vhids. Also, a common error is to have (slightly) different ip's, netmasks or aliases on the the carp interfaces for the two hosts. Not disputing the fact that I have done something wrong, but perhaps my reply should have been more succinct, in that: I tried with different VHIDs and the error was the same, ie, CARP still worked, however it did not increase the advskew on all carp interfaces on the same host when one carp interface was taken off line preventing the backup firewall from preempting all interfaces. To clarify, CARP is working in terms of redundancy, what does not seem to be working is the preempting of the primary firewall interfaces by the backup firewall should _one_ of the primary interfaces be taken off line. I returned the interfaces to carp0 = VHID 1 and carp1 = VHID 2 on both firewalls... still the same preempting problem. Mikel
Re: Can't get carp to fail over all interfaces with pfsync
Hi, On Tue, 10.11.2009 at 19:53:40 +1100, Mikel Lindsaar raasd...@gmail.com wrote: To clarify, CARP is working in terms of redundancy, what does not seem to be working is the preempting of the primary firewall interfaces by the backup firewall should _one_ of the primary interfaces be taken off line. Did you set the appropriate sysctl switch? net.inet.carp.preempt=1 Kind regards, --Toni++
IGNORE: Re: Can't get carp to fail over all interfaces with pfsync
On Tue, 10.11.2009 at 13:58:26 +0100, Toni Mueller openbsd-m...@oeko.net wrote: Did you set the appropriate sysctl switch? net.inet.carp.preempt=1 Note to self: Don't write emails when not fully awake. -- Kind regards, --Toni++
Re: Can't get carp to fail over all interfaces with pfsync
On Tue, Nov 10, 2009 at 8:09 PM, Camiel Dobbelaar c...@sentia.nl wrote: To clarify, CARP is working in terms of redundancy, what does not seem to be working is the preempting of the primary firewall interfaces by the backup firewall should _one_ of the primary interfaces be taken off line Use carpdemote. (man ifconfig and see also /etc/rc) Yes, that works. I think the FAQ needs updating then. This part specifically does not seem correct: http://www.openbsd.org/faq/pf/carp.html#forcefail If you take the physical interface down with ifconfig, then pfsync will take all the other carp interfaces and take them off line. If you take a carp interface off line, then pfsync will not take the other carp interfaces off line. To clarify. Primary firewall with two carp interfaces. Backup firewall with the same two carp interfaces. If you do: Primrary Firewall # ifconfig carp1 down then the result is: Primary Firewall: carp0 MASTER carp1 INIT Backup Firewall: carp0 BACKUP carp1 MASTER And no traffic flows. If you do: Primrary Firewall # ifconfig vr1 down (vr1 is the interface carp1 is on) then the result is: Primary Firewall: carp0 BACKUP carp1 INIT Backup Firewall: carp0 MASTER carp1 MASTER So that works as expected. I will write a change for the FAQ. Mikel
Can't get carp to fail over all interfaces with pfsync
Hi list,
So googled, went through http://www.openbsd.org/faq/pf/carp.html a few times
as well as the archives including one large thread which seemed to deal with
this exact issue, but the solution was setting the VHID to the same on all
carp interfaces (which I have already tried), and I can't see where I am
screwing up.
CARP works, in terms of if I take one router down, the other router becomes
master and when the first router comes back online, it preempts the master
role back to itself. This is expected behaviour and works fine, I can
reboot routers with impunity.
What is not working, is if I stand on the master firewall, and ifconfig
carp0 down, then the carp0 goes into INIT, and the backup firewall carp0
goes into MASTER, however, the primary firewall carp1 still stays MASTER and
the backup carp1 stays as BACKUP. As a consequence, traffic does not flow
across the routers as you end up with:
FW1 CARP0 - INIT
FW1 CARP1 - MASTER
FW2 CARP0 - MASTER
FW2 CARP1 - BACKUP
If I then ifconfig carp1 down on the master firewall I get:
FW1 CARP0 - INIT
FW1 CARP1 - INIT
FW2 CARP0 - MASTER
FW2 CARP1 - MASTER
And traffic flows again.
This seems contrary to http://www.openbsd.org/faq/pf/carp.html which states
if you init one interface, then all carp interfaces on that redundancy group
will advertise an infinite advskew.
I have a pair of Soekris Net5501 routers with the following setup:
+| WAN/Internet |+
||
|vr0| |vr0|
+-+ +-+
| fw1 |-vr3--vr3-| fw2 |
+-+ +-+
||
|trunk1| |trunk1|
||
---+---Shared LAN---+---
Trunk1 on both routers are two NICs (vr1 vr2) bonded in a trunk group
Both routers are running 4.6 GENERIC#58 i386
On both firewalls, in pf.conf there is:
# Top of pf.conf is:
pfsync_if=vr3
carp_ext_if=carp0
carp_int_if=carp1
carpdevs={ vr0 vr1 vr2 carp0_ext_if carp1_ext_if }
# .. skip tables, rdr, nat etc ...
#near the top of the ruleset is:
set skip on lo
set skip on $pfsync_if
pass quick on $carpdevs proto carp
On both firewalls sysctl for carp is:
$ sysctl | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2
FW1 hostname.if files are:
$ cat /etc/hostname.carp0
inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass
password
$ cat /etc/hostname.carp1
inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass
password
$ cat /etc/hostname.pfsync0
up syncdev vr3
$ cat /etc/hostname.vr0
inet 192.168.167.52 255.255.255.248 NONE
$ cat /etc/hostname.vr1
up
$ cat /etc/hostname.vr2
up
$ cat /etc/hostname.vr3
inet 172.16.0.1 255.255.255.252 NONE
FW2 hostname.if files are:
$ cat /etc/hostname.carp0
inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 128
pass password
$ cat /etc/hostname.carp1
inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 128
pass password
$ cat /etc/hostname.pfsync0
up syncdev vr3
$ cat /etc/hostname.vr0
inet 192.168.167.53 255.255.255.248
$ cat /etc/hostname.vr1
up
$ cat /etc/hostname.vr2
up
$ cat /etc/hostname.vr3
inet 172.16.0.2 255.255.255.252 NONE
Netstat Returns:
fw1 $ netstat -s -p carp
carp:
34 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
0 discarded for unknown vhid
0 discarded because of a bad address list
580 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
2 transitions to master
fw1 $ netstat -s -p pfsync
pfsync:
378 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for bad ttl
0 packets shorter than header
0 packets discarded for bad version
0 packets discarded for bad HMAC
0 packets discarded for bad action
0 packets discarded for short packet
0 states discarded for bad values
0 stale states
290 failed state lookup/inserts
488 packets sent (IPv4)
0 packets sent (IPv6)
0 send failed due to mbuf memory error
0 send error
fw2 $ netstat -s -p carp
carp:
799 packets received (IPv4)
0 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter
Re: Can't get carp to fail over all interfaces with pfsync
FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass password $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass password $ cat /etc/hostname.pfsync0 Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one?
Re: Can't get carp to fail over all interfaces with pfsync
On Tue, Nov 10, 2009 at 5:37 PM, Daniel Ouellet dan...@presscom.net wrote: FW1 hostname.if files are: $ cat /etc/hostname.carp0 inet 192.168.167.54 255.255.255.248 192.168.167.55 vhid 1 advskew 0 pass password $ cat /etc/hostname.carp1 inet 192.168.110.254 255.255.255.224 192.168.110.255 vhid 1 advskew 0 pass password $ cat /etc/hostname.pfsync0 Shouldn't you run different vhid ID of carp on different carp instance. Here you have Carp0 and carp 1 both running with vhid 1, so how will the system see them as different one? Initially I had them running as different VHIDs. carp0 was vhid 1 and carp1 was vhid 2, however, this did not work either... plus I would get unknown vhid errors in the netstat -s -p pfsync output if I had different vhids. Mikel
