Re: NAT on CARP interface
> Nevertheless, writing egress or $ext_If, what difference does it really
> make? You're just repeating a different word. Lol
It doesn't make any difference for me.
Being curious I added em0 do egress group and restarted all intefaces. However,
em0 seems not to be in ergess group and the rule with egress still doesn't
work:
match out log on egress from $int_if:network to any nat-to $ext_carpif
# cat /etc/hostname.em0
-inet
group egress
up
# ifconfig em0
em0: flags=8b43 mtu
1500
lladdr 00:0d:b9:59:e0:90
index 1 priority 0 llprio 3
media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
status: active
# ifconfig egress
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:01
description: WAN_KRZ
index 7 priority 15 llprio 3
carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 0
groups: carp egress
status: master
inet 10.0.15.216 netmask 0xff00 broadcast 10.0.15.255
# ifconfig carp0
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:01
description: WAN_KRZ
index 7 priority 15 llprio 3
carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 0
groups: carp egress
status: master
inet 10.0.15.216 netmask 0xff00 broadcast 10.0.15.255
> Does that rule you posted error out or are you just seeing blocks with it?
Just seeing blocks.
> https://www.openbsd.org/faq/pf/filter.html#syntax
>
> "The egress group, which contains the interface(s) that holds the default
> route(s)."
So.. carp0 contains default route, carp0 is in egress group.
carp0 refers to em0.
But...
match out log on carp0... - doesn't work
match out log on egress... - doesn't work
match out log on em0... - works!
I dont know...
On Sun, 28 Apr 2024 13:44:05 -0400
Mike wrote:
> Oh now I remember, you might need to add it to the egress interface group.
>
> Does that rule you posted error out or are you just seeing blocks with it?
>
>
> On Sun, Apr 28, 2024, 12:49 PM Mike wrote:
>
> > If I remember right, you can run 'ifconfig' and see if that interface is
> > marked as an egress interface or not. I can't remember how OBSD determines
> > what interfaces are egress or not but your em0 seems to be in a private
> > network so it might not be classifying itself as egress.
> >
> > Nevertheless, writing egress or $ext_If, what difference does it really
> > make? You're just repeating a different word. Lol
> >
> > On Sun, Apr 28, 2024, 12:08 PM Radek wrote:
> >
> >> > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it
> >> will work as the rule you say works.
> >> I made minor changes and tested the egress version.
> >>
> >> ext_if = "em0"
> >> ext_carpif = "carp0"
> >> int_if = "carp2"
> >> This rule works for me:
> >> match out log on $ext_if from $int_if:network to any nat-to $ext_carpif
> >>
> >> It seems it should work fine as well but it doesn't:
> >> match out log on egress from $int_if:network to any nat-to $ext_carpif
> >>
> >>
> >> On Thu, 25 Apr 2024 13:53:32 -0700
> >> obs...@loopw.com wrote:
> >>
> >> >
> >> >
> >> > > On Apr 25, 2024, at 10:36 AM, Radek wrote:
> >> > >
> >> > > Thank you for all your hints.
> >> > >
> >> > >> match out on egress from $lan_if:network to any nat-to (egress:0)
> >> > > This rule doesn't work.
> >> >
> >> > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it
> >> will work as the rule you say works.
> >> >
> >> >
> >> > fwiw, the $lan_if came from your configs existing “match”
> >> >
> >> > https://www.openbsd.org/faq/pf/filter.html#syntax - under “interface”
> >> you can find out about “egress”. I definitely prefer it to hard coding an
> >> interface in yet another line of a pf.conf
> >> >
> >> > I was presuming you didnt mind matching to $ext_if’s ip for new
> >> sessions outbound, hence (egress:0). Matching to the carp ip works. (this
> >> is basically a source nat rule in commercial-network-vendor speak)
> >> >
> >> >
> >> > >
> >> > >> ext_if=em0
> >> > >> int_if=vlan2
> >> > >> ext_carpIf=carp0
> >> >
> >> > >> match out on $ext_if inet from $int_if:network to any nat-to
> >> $ext_carpIf
> >> > > This rule works as expected.
> >> >
> >>
> >>
> >> Radek
> >>
> >>
Radek
Re: NAT on CARP interface
Oh now I remember, you might need to add it to the egress interface group. Does that rule you posted error out or are you just seeing blocks with it? On Sun, Apr 28, 2024, 12:49 PM Mike wrote: > If I remember right, you can run 'ifconfig' and see if that interface is > marked as an egress interface or not. I can't remember how OBSD determines > what interfaces are egress or not but your em0 seems to be in a private > network so it might not be classifying itself as egress. > > Nevertheless, writing egress or $ext_If, what difference does it really > make? You're just repeating a different word. Lol > > On Sun, Apr 28, 2024, 12:08 PM Radek wrote: > >> > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it >> will work as the rule you say works. >> I made minor changes and tested the egress version. >> >> ext_if = "em0" >> ext_carpif = "carp0" >> int_if = "carp2" >> This rule works for me: >> match out log on $ext_if from $int_if:network to any nat-to $ext_carpif >> >> It seems it should work fine as well but it doesn't: >> match out log on egress from $int_if:network to any nat-to $ext_carpif >> >> >> On Thu, 25 Apr 2024 13:53:32 -0700 >> obs...@loopw.com wrote: >> >> > >> > >> > > On Apr 25, 2024, at 10:36 AM, Radek wrote: >> > > >> > > Thank you for all your hints. >> > > >> > >> match out on egress from $lan_if:network to any nat-to (egress:0) >> > > This rule doesn't work. >> > >> > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it >> will work as the rule you say works. >> > >> > >> > fwiw, the $lan_if came from your configs existing “match” >> > >> > https://www.openbsd.org/faq/pf/filter.html#syntax - under “interface” >> you can find out about “egress”. I definitely prefer it to hard coding an >> interface in yet another line of a pf.conf >> > >> > I was presuming you didnt mind matching to $ext_if’s ip for new >> sessions outbound, hence (egress:0). Matching to the carp ip works. (this >> is basically a source nat rule in commercial-network-vendor speak) >> > >> > >> > > >> > >> ext_if=em0 >> > >> int_if=vlan2 >> > >> ext_carpIf=carp0 >> > >> > >> match out on $ext_if inet from $int_if:network to any nat-to >> $ext_carpIf >> > > This rule works as expected. >> > >> >> >> Radek >> >>
Re: NAT on CARP interface
On Sun, Apr 28, 2024, at 11:49 AM, Mike wrote: > If I remember right, you can run 'ifconfig' and see if that interface > is marked as an egress interface or not. I can't remember how OBSD > determines what interfaces are egress or not but your em0 seems to be https://www.openbsd.org/faq/pf/filter.html#syntax "The egress group, which contains the interface(s) that holds the default route(s)." > in a private network so it might not be classifying itself as egress. > > Nevertheless, writing egress or $ext_If, what difference does it really > make? You're just repeating a different word. Lol Brian Conway Owner RCE Software, LLC
Re: NAT on CARP interface
If I remember right, you can run 'ifconfig' and see if that interface is marked as an egress interface or not. I can't remember how OBSD determines what interfaces are egress or not but your em0 seems to be in a private network so it might not be classifying itself as egress. Nevertheless, writing egress or $ext_If, what difference does it really make? You're just repeating a different word. Lol On Sun, Apr 28, 2024, 12:08 PM Radek wrote: > > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it will > work as the rule you say works. > I made minor changes and tested the egress version. > > ext_if = "em0" > ext_carpif = "carp0" > int_if = "carp2" > This rule works for me: > match out log on $ext_if from $int_if:network to any nat-to $ext_carpif > > It seems it should work fine as well but it doesn't: > match out log on egress from $int_if:network to any nat-to $ext_carpif > > > On Thu, 25 Apr 2024 13:53:32 -0700 > obs...@loopw.com wrote: > > > > > > > > On Apr 25, 2024, at 10:36 AM, Radek wrote: > > > > > > Thank you for all your hints. > > > > > >> match out on egress from $lan_if:network to any nat-to (egress:0) > > > This rule doesn't work. > > > > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it will > work as the rule you say works. > > > > > > fwiw, the $lan_if came from your configs existing “match” > > > > https://www.openbsd.org/faq/pf/filter.html#syntax - under “interface” > you can find out about “egress”. I definitely prefer it to hard coding an > interface in yet another line of a pf.conf > > > > I was presuming you didnt mind matching to $ext_if’s ip for new sessions > outbound, hence (egress:0). Matching to the carp ip works. (this is > basically a source nat rule in commercial-network-vendor speak) > > > > > > > > > >> ext_if=em0 > > >> int_if=vlan2 > > >> ext_carpIf=carp0 > > > > >> match out on $ext_if inet from $int_if:network to any nat-to > $ext_carpIf > > > This rule works as expected. > > > > > Radek > >
Re: NAT on CARP interface
> change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it will work > as the rule you say works. I made minor changes and tested the egress version. ext_if = "em0" ext_carpif = "carp0" int_if = "carp2" This rule works for me: match out log on $ext_if from $int_if:network to any nat-to $ext_carpif It seems it should work fine as well but it doesn't: match out log on egress from $int_if:network to any nat-to $ext_carpif On Thu, 25 Apr 2024 13:53:32 -0700 obs...@loopw.com wrote: > > > > On Apr 25, 2024, at 10:36 AM, Radek wrote: > > > > Thank you for all your hints. > > > >> match out on egress from $lan_if:network to any nat-to (egress:0) > > This rule doesn't work. > > change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it will work > as the rule you say works. > > > fwiw, the $lan_if came from your configs existing “match” > > https://www.openbsd.org/faq/pf/filter.html#syntax - under “interface” you > can find out about “egress”. I definitely prefer it to hard coding an > interface in yet another line of a pf.conf > > I was presuming you didnt mind matching to $ext_if’s ip for new sessions > outbound, hence (egress:0). Matching to the carp ip works. (this is > basically a source nat rule in commercial-network-vendor speak) > > > > > >> ext_if=em0 > >> int_if=vlan2 > >> ext_carpIf=carp0 > > >> match out on $ext_if inet from $int_if:network to any nat-to $ext_carpIf > > This rule works as expected. > Radek
Re: NAT on CARP interface
> On Apr 25, 2024, at 10:36 AM, Radek wrote: > > Thank you for all your hints. > >> match out on egress from $lan_if:network to any nat-to (egress:0) > This rule doesn't work. change $lan_if to $int_if, change (egress:0) to $ext_carpif, and it will work as the rule you say works. fwiw, the $lan_if came from your configs existing “match” https://www.openbsd.org/faq/pf/filter.html#syntax - under “interface” you can find out about “egress”. I definitely prefer it to hard coding an interface in yet another line of a pf.conf I was presuming you didnt mind matching to $ext_if’s ip for new sessions outbound, hence (egress:0). Matching to the carp ip works. (this is basically a source nat rule in commercial-network-vendor speak) > >> ext_if=em0 >> int_if=vlan2 >> ext_carpIf=carp0 >> match out on $ext_if inet from $int_if:network to any nat-to $ext_carpIf > This rule works as expected.
Re: NAT on CARP interface
Thank you for all your hints.
> match out on egress from $lan_if:network to any nat-to (egress:0)
This rule doesn't work.
> ext_if=em0
> int_if=vlan2
> ext_carpIf=carp0
> match out on $ext_if inet from $int_if:network to any nat-to $ext_carpIf
This rule works as expected.
On Wed, 24 Apr 2024 17:14:49 -0400
Mike wrote:
> This command should help but you may need to add some "log" to your rules:
>
> tcpdump -nettti pflog0 will probably tell you.
>
> I don't have a bsd VM around to test but your int_if and ext_if should
> still refer to the underlying interface, not the carp.
>
> I'd change:
>
> ext_if=em0
> int_if=vlan2
> ext_carpIf=carp0
>
> match out on $ext_if inet from 10.0.2.0/24 to any nat-to $ext_carpIf
>
>
>
>
>
>
> On Wed, Apr 24, 2024, 4:50 PM Radek wrote:
>
> > Hi everyone,
> > it's a lab, the goal is a redundant firewalls with CARP and PFSYNC, I'm
> > trying to configure the master box. On the LAN side I have created carp2 on
> > vlan2 interface and it works as expected.
> > On the WAN side I can't figure out how to make NAT work on carp0 interface.
> > Can someone tell me where I have the wrong or missing configuration?
> >
> > OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
> > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >
> > # cat /etc/hostname.em1
> > -inet
> > up
> >
> > # cat /etc/hostname.vlan2
> > -inet
> > vnetid 2 parent em1 description "Interface VLAN-KRZ_LAN" up
> >
> > # cat /etc/hostname.carp2
> > -inet
> > inet 10.0.2.254 255.255.255.0 NONE vhid 2 advbase 1 advskew 0 carpdev
> > vlan2 pass test54321
> >
> >
> > # cat /etc/hostname.em0
> > -inet
> > up
> >
> > # cat /etc/hostname.carp0
> > -inet
> > inet 10.0.15.216 255.255.255.0 NONE description "WAN_KRZ" vhid 1 advbase 1
> > advskew 0 carpdev em0 pass test678
> >
> >
> > # cat /etc/pf.conf
> > ext_if = "carp0"
> > lan_if = "carp2"
> > pfsync_if = "em3"
> > internal_if = "vlan1010"
> > set skip on { lo0 vlan em3}
> > # pfsync and carp
> > pass quick on { $pfsync_if } proto pfsync #keep state (no-sync)
> > pass on { $internal_if } proto carp keep state (no-sync)
> > # nat
> > match out on $ext_if from $lan_if:network to any nat-to $ext_if
> > pass out
> >
> > # pfctl -s rules
> > pass quick on em3 proto pfsync all
> > pass on vlan1010 proto carp all keep state (no-sync)
> > match out on carp0 inet from 10.0.2.0/24 to any nat-to 10.0.15.216
> > pass out all flags S/SA
> >
> > # route -n show
> > Routing tables
> >
> > Internet:
> > DestinationGatewayFlags Refs Use Mtu Prio
> > Iface
> > 224/4 127.0.0.1 URS0 72 32768 8 lo0
> > 10.0.2/24 10.0.2.254 UCn10 -19
> > carp2
> > 10.0.2.201 18:03:73:b4:fa:c1 UHLc 011815 -18
> > carp2
> > 10.0.2.254 00:00:5e:00:01:02 UHLl 0 36 - 1
> > carp2
> > 10.0.2.255 10.0.2.254 UHb04 - 1
> > carp2
> > [snip]
> >
> > Radek
> >
> >
Radek
Re: NAT on CARP interface
try using egress and not carp0 on your match out rule. you’re not technically
sending out on the carp interface, only receiving.
make your match rule like this:
match out on egress from $lan_if:network to any nat-to (egress:0)
> On Apr 24, 2024, at 11:05 AM, Radek wrote:
>
> Hi everyone,
> it's a lab, the goal is a redundant firewalls with CARP and PFSYNC, I'm
> trying to configure the master box. On the LAN side I have created carp2 on
> vlan2 interface and it works as expected.
> On the WAN side I can't figure out how to make NAT work on carp0 interface.
> Can someone tell me where I have the wrong or missing configuration?
>
> OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
>dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> # cat /etc/hostname.em1
> -inet
> up
>
> # cat /etc/hostname.vlan2
> -inet
> vnetid 2 parent em1 description "Interface VLAN-KRZ_LAN" up
>
> # cat /etc/hostname.carp2
> -inet
> inet 10.0.2.254 255.255.255.0 NONE vhid 2 advbase 1 advskew 0 carpdev vlan2
> pass test54321
>
>
> # cat /etc/hostname.em0
> -inet
> up
>
> # cat /etc/hostname.carp0
> -inet
> inet 10.0.15.216 255.255.255.0 NONE description "WAN_KRZ" vhid 1 advbase 1
> advskew 0 carpdev em0 pass test678
>
>
> # cat /etc/pf.conf
> ext_if = "carp0"
> lan_if = "carp2"
> pfsync_if = "em3"
> internal_if = "vlan1010"
> set skip on { lo0 vlan em3}
> # pfsync and carp
> pass quick on { $pfsync_if } proto pfsync #keep state (no-sync)
> pass on { $internal_if } proto carp keep state (no-sync)
> # nat
> match out on $ext_if from $lan_if:network to any nat-to $ext_if
> pass out
>
> # pfctl -s rules
> pass quick on em3 proto pfsync all
> pass on vlan1010 proto carp all keep state (no-sync)
> match out on carp0 inet from 10.0.2.0/24 to any nat-to 10.0.15.216
> pass out all flags S/SA
>
> # route -n show
> Routing tables
>
> Internet:
> DestinationGatewayFlags Refs Use Mtu Prio Iface
> 224/4 127.0.0.1 URS0 72 32768 8 lo0
> 10.0.2/24 10.0.2.254 UCn10 -19 carp2
> 10.0.2.201 18:03:73:b4:fa:c1 UHLc 011815 -18 carp2
> 10.0.2.254 00:00:5e:00:01:02 UHLl 0 36 - 1 carp2
> 10.0.2.255 10.0.2.254 UHb04 - 1 carp2
> [snip]
>
> Radek
Re: NAT on CARP interface
This command should help but you may need to add some "log" to your rules:
tcpdump -nettti pflog0 will probably tell you.
I don't have a bsd VM around to test but your int_if and ext_if should
still refer to the underlying interface, not the carp.
I'd change:
ext_if=em0
int_if=vlan2
ext_carpIf=carp0
match out on $ext_if inet from 10.0.2.0/24 to any nat-to $ext_carpIf
On Wed, Apr 24, 2024, 4:50 PM Radek wrote:
> Hi everyone,
> it's a lab, the goal is a redundant firewalls with CARP and PFSYNC, I'm
> trying to configure the master box. On the LAN side I have created carp2 on
> vlan2 interface and it works as expected.
> On the WAN side I can't figure out how to make NAT work on carp0 interface.
> Can someone tell me where I have the wrong or missing configuration?
>
> OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> # cat /etc/hostname.em1
> -inet
> up
>
> # cat /etc/hostname.vlan2
> -inet
> vnetid 2 parent em1 description "Interface VLAN-KRZ_LAN" up
>
> # cat /etc/hostname.carp2
> -inet
> inet 10.0.2.254 255.255.255.0 NONE vhid 2 advbase 1 advskew 0 carpdev
> vlan2 pass test54321
>
>
> # cat /etc/hostname.em0
> -inet
> up
>
> # cat /etc/hostname.carp0
> -inet
> inet 10.0.15.216 255.255.255.0 NONE description "WAN_KRZ" vhid 1 advbase 1
> advskew 0 carpdev em0 pass test678
>
>
> # cat /etc/pf.conf
> ext_if = "carp0"
> lan_if = "carp2"
> pfsync_if = "em3"
> internal_if = "vlan1010"
> set skip on { lo0 vlan em3}
> # pfsync and carp
> pass quick on { $pfsync_if } proto pfsync #keep state (no-sync)
> pass on { $internal_if } proto carp keep state (no-sync)
> # nat
> match out on $ext_if from $lan_if:network to any nat-to $ext_if
> pass out
>
> # pfctl -s rules
> pass quick on em3 proto pfsync all
> pass on vlan1010 proto carp all keep state (no-sync)
> match out on carp0 inet from 10.0.2.0/24 to any nat-to 10.0.15.216
> pass out all flags S/SA
>
> # route -n show
> Routing tables
>
> Internet:
> DestinationGatewayFlags Refs Use Mtu Prio
> Iface
> 224/4 127.0.0.1 URS0 72 32768 8 lo0
> 10.0.2/24 10.0.2.254 UCn10 -19
> carp2
> 10.0.2.201 18:03:73:b4:fa:c1 UHLc 011815 -18
> carp2
> 10.0.2.254 00:00:5e:00:01:02 UHLl 0 36 - 1
> carp2
> 10.0.2.255 10.0.2.254 UHb04 - 1
> carp2
> [snip]
>
> Radek
>
>
NAT on CARP interface
Hi everyone,
it's a lab, the goal is a redundant firewalls with CARP and PFSYNC, I'm trying
to configure the master box. On the LAN side I have created carp2 on vlan2
interface and it works as expected.
On the WAN side I can't figure out how to make NAT work on carp0 interface.
Can someone tell me where I have the wrong or missing configuration?
OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
# cat /etc/hostname.em1
-inet
up
# cat /etc/hostname.vlan2
-inet
vnetid 2 parent em1 description "Interface VLAN-KRZ_LAN" up
# cat /etc/hostname.carp2
-inet
inet 10.0.2.254 255.255.255.0 NONE vhid 2 advbase 1 advskew 0 carpdev vlan2
pass test54321
# cat /etc/hostname.em0
-inet
up
# cat /etc/hostname.carp0
-inet
inet 10.0.15.216 255.255.255.0 NONE description "WAN_KRZ" vhid 1 advbase 1
advskew 0 carpdev em0 pass test678
# cat /etc/pf.conf
ext_if = "carp0"
lan_if = "carp2"
pfsync_if = "em3"
internal_if = "vlan1010"
set skip on { lo0 vlan em3}
# pfsync and carp
pass quick on { $pfsync_if } proto pfsync #keep state (no-sync)
pass on { $internal_if } proto carp keep state (no-sync)
# nat
match out on $ext_if from $lan_if:network to any nat-to $ext_if
pass out
# pfctl -s rules
pass quick on em3 proto pfsync all
pass on vlan1010 proto carp all keep state (no-sync)
match out on carp0 inet from 10.0.2.0/24 to any nat-to 10.0.15.216
pass out all flags S/SA
# route -n show
Routing tables
Internet:
DestinationGatewayFlags Refs Use Mtu Prio Iface
224/4 127.0.0.1 URS0 72 32768 8 lo0
10.0.2/24 10.0.2.254 UCn10 -19 carp2
10.0.2.201 18:03:73:b4:fa:c1 UHLc 011815 -18 carp2
10.0.2.254 00:00:5e:00:01:02 UHLl 0 36 - 1 carp2
10.0.2.255 10.0.2.254 UHb04 - 1 carp2
[snip]
Radek
Re: CARP and VRRP compliance
Hello Theo, It's disheartening to see the disparity in treatment between entities like OpenBSD and larger corporations within these governance structures. However, your resolve in the face of such challenges is commendable. The creation of CARP, under the circumstances you described, not only serves as a practical solution but also as a principled stand against the monopolization of technology standards. This unwavering commitment is the reason OpenBSD is so deeply respected and cherished. Thank you for your perseverance and for setting an example of integrity in the technology community. This is why we love OpenBSD so much. Kind regards Sam On Wed, Feb 14, 2024 at 7:26 PM Theo de Raadt wrote: > Stuart Henderson wrote: > > > On 2024-02-13, Samuel Jayden wrote: > > > From the information provided in the link, it appears that CARP and > VRRP > > > protocols aren't inherently interoperable. > > > > They are different protocols - they *had* to be different because VRRP > > was subject to patents. And if carp was changed now, it wouldn't be > > interoperable with existing carp installations. > > > > > While Cisco may have attempted to address this by introducing a command > > > like "disable-loop-detection carp" in its Nexus 1000V virtual router > > > product, this solution unfortunately doesn't extend to standard router > > > hardware, rendering it ineffective in many scenarios. > > > > That's not about interop beteeen carp and vrrp speakers, it's about > > using carp (or vrrp or hsrp or similar) on a port attached to the > > 'virtual switch'. See 'Information About Redundant Routing Protocols' on > > > https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html > > > > > Is it feasible to achieve CARP and VRRP interoperability through a > > > user-space application? > > > > No. They are different protocols. For what you want to do, running VRRP > > on the OpenBSD box might make some sense though. There are various > > existing userland implementations of VRRP that might be able to run > > on OpenBSD, probably with some work to port them - e.g. freevrrpd, > > frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted > > to try I'd suggest starting by looking at freevrrpd). > > This was my experience: > > VRRP was the first patent-encumbered protocol squeezed through the IETF > process. > > The backers of that change in process were employees and laywers at a few > major companies, but also tightly integrated into the IETF approval > process. > > When we objected to the VRRP situation, they circled the wagons, not just > to defend the VRRP patent, but to protect a future of patent's being OK in > IETF processes. > > In response, OpenBSD carefully developed a similar mechanism called CARP, > and the acronymn actually expands to "Cisco Asshole Redundancy Protocol", > because the main traitors inside IETF were Cisco employees. > > Then we asked IETF for numbers to make this a unique protocol. Unlike > a recent threads where Tatu asked IETF for port 22 and they just gave it > to him, the various number authorities inside IETF demanded that we follow > the most stringent procedures for CARP. Even to this day, IETF provides > the various prototol numbers to some large corporate industry members > without > forcing them down those stringent procedures. > > As a result, we simply squatted on the VRRP numbers. We gave them plenty > of warning we would be doing this. Over the following years, we heard some > real anger IETF decision makers internally, but none of them re-visited our > request for seperate numbers. We never got numbers. So CARP will stay > where it is. > > One major bug was in VRRP on some HP product was found in the first year. > CARP packets were incorrectly parsed as VRRP packets. I don't remember > the details, but I think it rebooted that HP device, probably a switch. > > Oh well. > >
Re: CARP and VRRP compliance
Greetings, I have now attained a deeper understanding of the topic at hand; thank you for your insights. It appears that my requirements necessitate communication between a Cisco router and VRRP, rather than CARP. Upon reviewing the open-source projects you've recommended, here are my findings: The vrrpd project seems quite distant from being readily compilable. It exhibits a classic Linux developer's perspective, showing no inclination towards ensuring compatibility with operating systems outside the Linux realm. I am still engaged with frr-vrrpd, yet, to my dismay, I haven't managed to compile it thus far. With freevrrpd, I am tantalizingly closer to a resolution. By crafting minor patches, I've successfully compiled it, albeit necessitating the deactivation of netgraph code. Upon conducting a VRRP test between OpenBSD + freevrrpd and a Cisco Router, I observed that both devices persisted in identifying themselves as the master. Monitoring the relevant interface with tcpdump allowed me to perceive packets emanating from the Cisco Router; however, there was a conspicuous absence of VRRP packets from the OpenBSD system. It seems plausible that disabling the netgraph code contributed to this predicament. Should there exist an equivalent to netgraph within OpenBSD, I am eager to explore that avenue. Thanks. Sam On Wed, Feb 14, 2024 at 2:06 PM Stuart Henderson wrote: > On 2024-02-13, Samuel Jayden wrote: > > From the information provided in the link, it appears that CARP and VRRP > > protocols aren't inherently interoperable. > > They are different protocols - they *had* to be different because VRRP > was subject to patents. And if carp was changed now, it wouldn't be > interoperable with existing carp installations. > > > While Cisco may have attempted to address this by introducing a command > > like "disable-loop-detection carp" in its Nexus 1000V virtual router > > product, this solution unfortunately doesn't extend to standard router > > hardware, rendering it ineffective in many scenarios. > > That's not about interop beteeen carp and vrrp speakers, it's about > using carp (or vrrp or hsrp or similar) on a port attached to the > 'virtual switch'. See 'Information About Redundant Routing Protocols' on > > https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html > > > Is it feasible to achieve CARP and VRRP interoperability through a > > user-space application? > > No. They are different protocols. For what you want to do, running VRRP > on the OpenBSD box might make some sense though. There are various > existing userland implementations of VRRP that might be able to run > on OpenBSD, probably with some work to port them - e.g. freevrrpd, > frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted > to try I'd suggest starting by looking at freevrrpd). > > -- > Please keep replies on the mailing list. > >
Re: CARP and VRRP compliance
Stuart Henderson wrote: > On 2024-02-13, Samuel Jayden wrote: > > From the information provided in the link, it appears that CARP and VRRP > > protocols aren't inherently interoperable. > > They are different protocols - they *had* to be different because VRRP > was subject to patents. And if carp was changed now, it wouldn't be > interoperable with existing carp installations. > > > While Cisco may have attempted to address this by introducing a command > > like "disable-loop-detection carp" in its Nexus 1000V virtual router > > product, this solution unfortunately doesn't extend to standard router > > hardware, rendering it ineffective in many scenarios. > > That's not about interop beteeen carp and vrrp speakers, it's about > using carp (or vrrp or hsrp or similar) on a port attached to the > 'virtual switch'. See 'Information About Redundant Routing Protocols' on > https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html > > > Is it feasible to achieve CARP and VRRP interoperability through a > > user-space application? > > No. They are different protocols. For what you want to do, running VRRP > on the OpenBSD box might make some sense though. There are various > existing userland implementations of VRRP that might be able to run > on OpenBSD, probably with some work to port them - e.g. freevrrpd, > frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted > to try I'd suggest starting by looking at freevrrpd). This was my experience: VRRP was the first patent-encumbered protocol squeezed through the IETF process. The backers of that change in process were employees and laywers at a few major companies, but also tightly integrated into the IETF approval process. When we objected to the VRRP situation, they circled the wagons, not just to defend the VRRP patent, but to protect a future of patent's being OK in IETF processes. In response, OpenBSD carefully developed a similar mechanism called CARP, and the acronymn actually expands to "Cisco Asshole Redundancy Protocol", because the main traitors inside IETF were Cisco employees. Then we asked IETF for numbers to make this a unique protocol. Unlike a recent threads where Tatu asked IETF for port 22 and they just gave it to him, the various number authorities inside IETF demanded that we follow the most stringent procedures for CARP. Even to this day, IETF provides the various prototol numbers to some large corporate industry members without forcing them down those stringent procedures. As a result, we simply squatted on the VRRP numbers. We gave them plenty of warning we would be doing this. Over the following years, we heard some real anger IETF decision makers internally, but none of them re-visited our request for seperate numbers. We never got numbers. So CARP will stay where it is. One major bug was in VRRP on some HP product was found in the first year. CARP packets were incorrectly parsed as VRRP packets. I don't remember the details, but I think it rebooted that HP device, probably a switch. Oh well.
Re: CARP and VRRP compliance
On 2024-02-13, Samuel Jayden wrote: > From the information provided in the link, it appears that CARP and VRRP > protocols aren't inherently interoperable. They are different protocols - they *had* to be different because VRRP was subject to patents. And if carp was changed now, it wouldn't be interoperable with existing carp installations. > While Cisco may have attempted to address this by introducing a command > like "disable-loop-detection carp" in its Nexus 1000V virtual router > product, this solution unfortunately doesn't extend to standard router > hardware, rendering it ineffective in many scenarios. That's not about interop beteeen carp and vrrp speakers, it's about using carp (or vrrp or hsrp or similar) on a port attached to the 'virtual switch'. See 'Information About Redundant Routing Protocols' on https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/layer_2_switching/configuration/guide/n1000v_l2/n1000v_l2_7redundantroutingprot.html > Is it feasible to achieve CARP and VRRP interoperability through a > user-space application? No. They are different protocols. For what you want to do, running VRRP on the OpenBSD box might make some sense though. There are various existing userland implementations of VRRP that might be able to run on OpenBSD, probably with some work to port them - e.g. freevrrpd, frr-vrrpd, vrrpd. Nothing already in the ports tree (if someone wanted to try I'd suggest starting by looking at freevrrpd). -- Please keep replies on the mailing list.
Re: CARP and VRRP compliance
Am 13.02.2024 19:07 schrieb Samuel Jayden: Also I've another question: Is it feasible to achieve CARP and VRRP interoperability through a user-space application? One step back.. you're looking for using one cisco router and one OpenBSD box as a redundant pair? I've no idea and in over 20y I did not consider doing this. If you think about how an OpenBSD pair (failover/load between themselves) and "on the other side" a Cisco pair using VRRP (acting betweeen themselves), I can tell that this works w/o having a stamped letter with some crayon on it. -- pb
Re: CARP and VRRP compliance
Hello Marcus, Thank you for your response. >From the information provided in the link, it appears that CARP and VRRP protocols aren't inherently interoperable. While Cisco may have attempted to address this by introducing a command like "disable-loop-detection carp" in its Nexus 1000V virtual router product, this solution unfortunately doesn't extend to standard router hardware, rendering it ineffective in many scenarios. Also I've another question: Is it feasible to achieve CARP and VRRP interoperability through a user-space application? I am curious if there are any existing solutions or approaches that leverage user-space applications to bridge the interoperability gap between CARP and VRRP. If anyone has insights or experiences in this area, I would greatly appreciate hearing about them. Thank you for considering my inquiries. Best regards Sam On Tue, Feb 13, 2024 at 8:26 PM Marcus MERIGHI wrote: > Hello Samuel, > > samueljaydan1...@gmail.com (Samuel Jayden), 2024.02.13 (Tue) 17:35 (CET): > > I am reaching out to seek guidance on creating redundancy between a Cisco > > Router and OpenBSD. After conducting extensive research on the subject, I > > find myself in need of clarification on a specific point. > > This has some background info for you: > > https://mwl.io/archives/1866 > > Marcus >
Re: CARP and VRRP compliance
Hello Samuel, samueljaydan1...@gmail.com (Samuel Jayden), 2024.02.13 (Tue) 17:35 (CET): > I am reaching out to seek guidance on creating redundancy between a Cisco > Router and OpenBSD. After conducting extensive research on the subject, I > find myself in need of clarification on a specific point. This has some background info for you: https://mwl.io/archives/1866 Marcus
CARP and VRRP compliance
Hello OpenBSD, I am reaching out to seek guidance on creating redundancy between a Cisco Router and OpenBSD. After conducting extensive research on the subject, I find myself in need of clarification on a specific point. My intention is to employ the use of the CARP protocol in OpenBSD and VRRP on the Cisco Router. However, I am uncertain about the compatibility between OpenBSD's CARP and Cisco's VRRP protocols. If any of you have practical experience or insights into using these two protocols simultaneously within the same broadcast domain, I would greatly appreciate hearing about your experiences. Thank you in advance for your time and assistance. Best regards Sam
Re: gre over carp
Yes, exactly. The fail-back never worked. All SAs would disappear and subsequent failover to slave dropped SAs there too. There was a story behind this from the developer somewhere in the mail archives. A while back. On Wednesday, December 13, 2023 at 05:55:50 p.m. GMT+9, Janne Johansson wrote: Den ons 13 dec. 2023 kl 04:15 skrev All : > > >I'd like to add sasyncd in the mix and a 2nd router for higher > availability. > Don't do it. sasyncd is known not to work properly in failover scenarios. > When I ran it it did work fine for the first fail-over, but seldom (or perhaps never?) on fail-back when the master returned, so it was ok for giving me nice redundancy if the current carp master died, then I could choose a suitable time after fixing this node when to take a the hit of a new tunnel-setup as I flipped back. Never knew why it would only work one-way for me, but we had certain issues with broadcom bnx(4) cards and multicast at the time, so it could have been related to that. > >Will gre over carp work? > I think you can just try out in a vm. Don't see the reason why it would > not. > But perhaps there are some features that CARP interface doesn't support > for gre. > Do mind that carp on software-defined switch networks might need some settings in order to allow the virtual eth cards to send out frames with "bogus" mac-addresses, since some hypervisors keep good track of which macs they have given to a VM and drop ethernet frames sent with not-those mac-addresses as source. But to add to the original reply, I would probably go for two gre's towards the non-carp ips, since you are using ospf anyhow, you might just tell ospfd that one gre has a slightly higher cost than the other, and let it deal with the new network map when one of them fails. -- May the most significant bit of your life be positive.
Re: gre over carp
Den ons 13 dec. 2023 kl 04:15 skrev All : > > >I'd like to add sasyncd in the mix and a 2nd router for higher > availability. > Don't do it. sasyncd is known not to work properly in failover scenarios. > When I ran it it did work fine for the first fail-over, but seldom (or perhaps never?) on fail-back when the master returned, so it was ok for giving me nice redundancy if the current carp master died, then I could choose a suitable time after fixing this node when to take a the hit of a new tunnel-setup as I flipped back. Never knew why it would only work one-way for me, but we had certain issues with broadcom bnx(4) cards and multicast at the time, so it could have been related to that. > >Will gre over carp work? > I think you can just try out in a vm. Don't see the reason why it would > not. > But perhaps there are some features that CARP interface doesn't support > for gre. > Do mind that carp on software-defined switch networks might need some settings in order to allow the virtual eth cards to send out frames with "bogus" mac-addresses, since some hypervisors keep good track of which macs they have given to a VM and drop ethernet frames sent with not-those mac-addresses as source. But to add to the original reply, I would probably go for two gre's towards the non-carp ips, since you are using ospf anyhow, you might just tell ospfd that one gre has a slightly higher cost than the other, and let it deal with the new network map when one of them fails. -- May the most significant bit of your life be positive.
Re: gre over carp
>I'd like to add sasyncd in the mix and a 2nd router for higher availability. Don't do it. sasyncd is known not to work properly in failover scenarios. >Will gre over carp work? I think you can just try out in a vm. Don't see the reason why it would not. But perhaps there are some features that CARP interface doesn't support for gre. On Tuesday, December 12, 2023 at 09:01:53 p.m. GMT+9, Kapetanakis Giannis wrote: I was wondering if anyone runs gre tunnels over carp. We have an OpenBSD router that does ipsec over gre tunnels with remote location routers in order to pass voip traffic. Routing is done with ospf on top of gre. I'd like to add sasyncd in the mix and a 2nd router for higher availability. Will gre over carp work? I'd prefer to use the carped IP in the tunnel and not the real IP of each server, so eventually 1 tunnel (active) per remote router. How will this work when I demote master and go via backup? Thanks, G
gre over carp
I was wondering if anyone runs gre tunnels over carp. We have an OpenBSD router that does ipsec over gre tunnels with remote location routers in order to pass voip traffic. Routing is done with ospf on top of gre. I'd like to add sasyncd in the mix and a 2nd router for higher availability. Will gre over carp work? I'd prefer to use the carped IP in the tunnel and not the real IP of each server, so eventually 1 tunnel (active) per remote router. How will this work when I demote master and go via backup? Thanks, G
Redundant carp mesh best practices
Hi, I have an ospf mesh running between routers 1, 2, 3 and 4. i have a bridge with a vether running on 1 and 2 each connecting to 3 and 4. i have a trunk running on 3 and 4 each connecting back to the previous 1 and 2. the reason for this setup is the bridge on 1 and 2 faciliates transferring the carp multicast frame between 3 and 4. I’m wondering if this is the current best practice or is there a better way? Image: https://i.redd.it/4i1lu4aeydzb1.jpg Thanks!
Re: carp flapping
Followup... On 5/12/23 08:17, Stuart Henderson wrote: On 2023-05-12, Nick Holland wrote: ... I had several other people suggest network problems. I'm not going to say "impossible" or even "unlikely", but my understanding is that the two machines are both plugged into the same switch, in the same rack. I've since had someone more familiar with the physical environment say my blind trust in their switch hw may be slightly misplaced. :) You can also look at netstat -ni -I ixl0 netstat -ni -I ixl0 -e kstat ixl0::: These looked REALLY clean. no drops, fails or collisions. which may give some other clues even pfctl -si might have something relevant Several people pointed out I was using the default advskew of 1 second, which means a small network glitch (or system load? maybe I'm all wrong about this system never breaking a sweat, at least when it comes to network traffic) would flip it, so I've increased it to 10 on both machines (and apparently just induced a flip of my own. oops). By the nature of this system, some people will be annoyed by any flip, so it really doesn't matter if it was a 1 second outage or a 30 second outage, I just want the system available again after an unhappy event (or routine maintenance). the course adjustment in seconds is advbase, advskew is a much smaller delay meant for a config with primary/backup where the backup advertises just slightly less frequently. Um. yeah. I set advbase, and typed advskew in the e-mail. my bad. After setting to 10, I have gone over two weeks without any flips, so that looks like that is a pretty good fix. Thanks for the guidance! Nick.
Re: carp flapping
On 16/05/2023 00:11, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote: Nick, spare yourself the pain and just designate one machine as the master. This is how we run all our proxy server pairs (nginx, squid, other stuff). For a pair fooa/foob, 'a' is the master, and gets advskew 100. The 'b' host gets 150. Make sure preemption is enabled. When it's upgrade time, upgrade the 'b' machine and reboot. If it looks stable, set its advskew to 50 and wait for it to pick up traffic. Now upgrade and reboot the 'a' host. When it looks happy, set 'b's advskew back to 150. This keeps everything in a known state. You are going to break connections no matter what -- even when you let the master float -- so you might as well do it under your own control. We schedule our updates for off-peak hours, and accept that the flip is going to interrupt traffic. You just have to live with it. We moved to this scheme on all our proxies and firewalls seven years ago and have never looked back. --lyndon Totally agree on this and on top of that add load balancers/routers in the mix which will run carp/relayd/pfsync/forwarding. With sticky sessions, all requests will redirect to the same backend server and you can avoid breaking service connections. These don't have to be big machines. G
Re: carp flapping
Nick, spare yourself the pain and just designate one machine as the master. This is how we run all our proxy server pairs (nginx, squid, other stuff). For a pair fooa/foob, 'a' is the master, and gets advskew 100. The 'b' host gets 150. Make sure preemption is enabled. When it's upgrade time, upgrade the 'b' machine and reboot. If it looks stable, set its advskew to 50 and wait for it to pick up traffic. Now upgrade and reboot the 'a' host. When it looks happy, set 'b's advskew back to 150. This keeps everything in a known state. You are going to break connections no matter what -- even when you let the master float -- so you might as well do it under your own control. We schedule our updates for off-peak hours, and accept that the flip is going to interrupt traffic. You just have to live with it. We moved to this scheme on all our proxies and firewalls seven years ago and have never looked back. --lyndon
Re: carp flapping
On 12/05/2023 14:43, Nick Holland wrote: > I had several other people suggest network problems. I'm not going to > say "impossible" or even "unlikely", but my understanding is that the > two machines are both plugged into the same switch, in the same rack. > > Several people pointed out I was using the default advskew of 1 second, > which means a small network glitch (or system load? maybe I'm all wrong > about this system never breaking a sweat, at least when it comes to > network traffic) would flip it, so I've increased it to 10 on both > machines (and apparently just induced a flip of my own. oops). By the > nature of this system, some people will be annoyed by any flip, so it > really doesn't matter if it was a 1 second outage or a 30 second outage, > I just want the system available again after an unhappy event (or > routine maintenance). > > Nick. Usually it's a network problem. The big delay of 3 days you had also suggests that. But on the other hand, I also had a similar problem in one of my load balancers (routing/fw/relayd), where the MASTER was becoming BACKUP for no obvious reason. I believed it was a network glitch, but couldn't trace it. The problem after all was that they where pushing the limit of max pf states and relayd checks where failing. Not obvious to spot at all. I believe default is 20K. pfctl -sm pfctl -si After increasing that limit with set limit states I've never had a glitch any more. G
Re: carp flapping
On 2023-05-12, Nick Holland wrote: > On 5/12/23 03:28, Stuart Henderson wrote: >> On 2023-05-12, Nick Holland wrote: >>> Here's the problem I've seen: I have my two machines flipping state >>> randomly(?). This bothers me because that means it is breaking people's >>> downloads. Longest period betweek flips was less than two weeks. >>> >>> So ... I cranked up the carp logging to 5 and then 7 to see what it had >>> to say about why...and it had almost nothing to say. >> >> Does netstat -s -p carp give any enlightenment? > > > ok, I just skewed the stats by taking the opportunity to bring the now > backup up to -current, so node1 does not have the most recent flap: > > node1 $ uptime > 7:18AM up 8:22, 1 user, load averages: 0.00, 0.05, 0.08 > > node1 $ doas netstat -s -p carp > carp: > 29981 packets received (IPv4) > 0 packets received (IPv6) > 0 packets discarded for bad interface > 0 packets discarded for wrong TTL > 0 packets shorter than header > 0 discarded for bad checksums > 0 discarded packets with a bad version > 0 discarded because packet too short > 0 discarded for bad authentication > 0 discarded for unknown vhid > 0 discarded because of a bad address list > 0 packets sent (IPv4) > 0 packets sent (IPv6) > 0 send failed due to mbuf memory error > 0 transitions to master > > node2 $ uptime > 7:19AM up 4 days, 20:58, 2 users, load averages: 0.83, 0.78, 0.73 > > $ ] netstat -s -p carp > carp: > 367836 packets received (IPv4) > 0 packets received (IPv6) > 0 packets discarded for bad interface > 0 packets discarded for wrong TTL > 0 packets shorter than header > 0 discarded for bad checksums > 0 discarded packets with a bad version > 0 discarded because packet too short > 0 discarded for bad authentication > 0 discarded for unknown vhid > 0 discarded because of a bad address list > 52806 packets sent (IPv4) > 0 packets sent (IPv6) > 0 send failed due to mbuf memory error > 2 transitions to master > > > Will monitor going forward, though. > > > I had several other people suggest network problems. I'm not going to > say "impossible" or even "unlikely", but my understanding is that the > two machines are both plugged into the same switch, in the same rack. You can also look at netstat -ni -I ixl0 netstat -ni -I ixl0 -e kstat ixl0::: which may give some other clues even pfctl -si might have something relevant > Several people pointed out I was using the default advskew of 1 second, > which means a small network glitch (or system load? maybe I'm all wrong > about this system never breaking a sweat, at least when it comes to > network traffic) would flip it, so I've increased it to 10 on both > machines (and apparently just induced a flip of my own. oops). By the > nature of this system, some people will be annoyed by any flip, so it > really doesn't matter if it was a 1 second outage or a 30 second outage, > I just want the system available again after an unhappy event (or > routine maintenance). the course adjustment in seconds is advbase, advskew is a much smaller delay meant for a config with primary/backup where the backup advertises just slightly less frequently.
Re: carp flapping
On 5/12/23 03:28, Stuart Henderson wrote: On 2023-05-12, Nick Holland wrote: Here's the problem I've seen: I have my two machines flipping state randomly(?). This bothers me because that means it is breaking people's downloads. Longest period betweek flips was less than two weeks. So ... I cranked up the carp logging to 5 and then 7 to see what it had to say about why...and it had almost nothing to say. Does netstat -s -p carp give any enlightenment? ok, I just skewed the stats by taking the opportunity to bring the now backup up to -current, so node1 does not have the most recent flap: node1 $ uptime 7:18AM up 8:22, 1 user, load averages: 0.00, 0.05, 0.08 node1 $ doas netstat -s -p carp carp: 29981 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 0 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 0 transitions to master node2 $ uptime 7:19AM up 4 days, 20:58, 2 users, load averages: 0.83, 0.78, 0.73 $ ] netstat -s -p carp carp: 367836 packets received (IPv4) 0 packets received (IPv6) 0 packets discarded for bad interface 0 packets discarded for wrong TTL 0 packets shorter than header 0 discarded for bad checksums 0 discarded packets with a bad version 0 discarded because packet too short 0 discarded for bad authentication 0 discarded for unknown vhid 0 discarded because of a bad address list 52806 packets sent (IPv4) 0 packets sent (IPv6) 0 send failed due to mbuf memory error 2 transitions to master Will monitor going forward, though. I had several other people suggest network problems. I'm not going to say "impossible" or even "unlikely", but my understanding is that the two machines are both plugged into the same switch, in the same rack. Several people pointed out I was using the default advskew of 1 second, which means a small network glitch (or system load? maybe I'm all wrong about this system never breaking a sweat, at least when it comes to network traffic) would flip it, so I've increased it to 10 on both machines (and apparently just induced a flip of my own. oops). By the nature of this system, some people will be annoyed by any flip, so it really doesn't matter if it was a 1 second outage or a 30 second outage, I just want the system available again after an unhappy event (or routine maintenance). Nick.
Re: carp flapping
On 2023-05-12, Nick Holland wrote: > Here's the problem I've seen: I have my two machines flipping state > randomly(?). This bothers me because that means it is breaking people's > downloads. Longest period betweek flips was less than two weeks. > > So ... I cranked up the carp logging to 5 and then 7 to see what it had > to say about why...and it had almost nothing to say. Does netstat -s -p carp give any enlightenment?
carp flapping
Hi, I have a couple identical servers that provide a few services (not FW or gateway -- http, ftp, etc.). Figured they would make a great CARP pair, so if the primary broke, the secondary would take over immediately. It would also make maintenance windows shorter...make changes on secondary machine, test, reboot primary to force the secondary to become master. The two machines should be equals. I have no preference on running on one machine or the other. IF nothing breaks, I'd prefer that the one that is serving keep serving until I tell it otherwise. Both machines should have no issue with performance with the tasks they have, lots of proc, lots of RAM, nvme disk, etc. Here's the problem I've seen: I have my two machines flipping state randomly(?). This bothers me because that means it is breaking people's downloads. Longest period betweek flips was less than two weeks. So ... I cranked up the carp logging to 5 and then 7 to see what it had to say about why...and it had almost nothing to say. Here is the info from messages from both machines for the most recent flip. Past ones look basically the same. Node 2: /var/log $ zgrep carp0 messages May 9 21:51:23 node2 /bsd: carp0: state transition: BACKUP -> MASTER May 9 21:51:25 node2 /bsd: carp0: state transition: MASTER -> BACKUP May 11 16:36:04 node2 /bsd: carp0: state transition: BACKUP -> MASTER Node 1: /var/log $ zgrep carp messages May 9 21:51:25 node1 /bsd: carp0: state transition: MASTER -> BACKUP May 9 21:51:28 node1 /bsd: carp0: state transition: BACKUP -> MASTER May 11 16:36:07 node1 /bsd: carp0: state transition: MASTER -> BACKUP hostname.carp0 from both machines: inet a.b.c.240 255.255.255.0 128.100.17.255 vhid 1 carpdev ixl0 pass censored inet alias a.b.c.241 255.255.255.255 128.100.17.255 inet alias a.b.c.243 255.255.255.255 128.100.17.255 inet alias a.b.c.246 255.255.255.255 128.100.17.255 verified identical (before slight anonymizing) on both systems. hostname.ixl0 on node1: inet a.b.c.248/24 hostname.ixl0 on node2: inet a.b.c.247 0xff00 pf.conf includes this before any other "quick" statements: pass quick inet proto carp all Is there something I'm missing? Incorrect expectations on my part? Nick. dmesg: OpenBSD 7.3-current (GENERIC.MP) #1175: Wed May 3 08:19:33 MDT 2023 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 50078154752 (47758MB) avail mem = 48540807168 (46292MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.2 @ 0x6f3c3000 (84 entries) bios0: vendor American Megatrends Inc. version "3.4" date 10/30/2020 bios0: Supermicro X11SPW-TF efi0 at bios0: UEFI 2.7 efi0: American Megatrends rev 0x5000e acpi0 at bios0: ACPI 6.2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP FPDT FIDT SPMI UEFI SSDT MCFG HPET APIC MIGT MSCT PCAT PCCT RASF SLIT SRAT SVOS WDDT OEM4 OEM1 SSDT OEM3 SSDT SSDT DMAR HEST BERT ERST EINJ WSMT acpi0: wakeup devices XHCI(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4) PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 acpimcfg0: addr 0x8000, bus 0-255 acpihpet0 at acpi0: 2399 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.06 MHz, 06-55-07 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line 16-way L2 cache, 8MB 64b/line 11-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 25MHz cpu0: mwait min=64, max=64, C-substates=0.2.0.2, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.09 MHz, 06-55-07 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP
Re: 7.3: high network latency every couple of seconds. Carp?
Please ignore this duplicate post and reply to the other thread on this mailing list. I had used my private EMail account by accident. Regards Harri
7.3: high network latency every couple of seconds. Carp?
Hi folks,
Using 7.3 on a HA gateway ("redgatea" and "redgateb", one external
network, 2 internal networks, carp on all interfaces) I see a high
network latency for incoming network traffic every couple of seconds.
Trying to ping redgatea from redgateb over the pfsync interface, for
example:
redgateb # ping 192.168.23.2
PING 192.168.23.2 (192.168.23.2): 56 data bytes
64 bytes from 192.168.23.2: icmp_seq=0 ttl=255 time=0.585 ms
64 bytes from 192.168.23.2: icmp_seq=1 ttl=255 time=48.559 ms
64 bytes from 192.168.23.2: icmp_seq=2 ttl=255 time=153.323 ms
64 bytes from 192.168.23.2: icmp_seq=3 ttl=255 time=0.233 ms
64 bytes from 192.168.23.2: icmp_seq=4 ttl=255 time=0.230 ms
64 bytes from 192.168.23.2: icmp_seq=5 ttl=255 time=0.227 ms
64 bytes from 192.168.23.2: icmp_seq=6 ttl=255 time=1.001 ms
64 bytes from 192.168.23.2: icmp_seq=7 ttl=255 time=1.253 ms
64 bytes from 192.168.23.2: icmp_seq=8 ttl=255 time=0.224 ms
64 bytes from 192.168.23.2: icmp_seq=9 ttl=255 time=0.229 ms
64 bytes from 192.168.23.2: icmp_seq=10 ttl=255 time=0.231 ms
64 bytes from 192.168.23.2: icmp_seq=11 ttl=255 time=0.228 ms
64 bytes from 192.168.23.2: icmp_seq=12 ttl=255 time=0.267 ms
64 bytes from 192.168.23.2: icmp_seq=13 ttl=255 time=259.893 ms
64 bytes from 192.168.23.2: icmp_seq=14 ttl=255 time=364.299 ms
64 bytes from 192.168.23.2: icmp_seq=15 ttl=255 time=0.228 ms
64 bytes from 192.168.23.2: icmp_seq=16 ttl=255 time=0.230 ms
64 bytes from 192.168.23.2: icmp_seq=17 ttl=255 time=0.231 ms
64 bytes from 192.168.23.2: icmp_seq=18 ttl=255 time=1.349 ms
64 bytes from 192.168.23.2: icmp_seq=19 ttl=255 time=1.113 ms
64 bytes from 192.168.23.2: icmp_seq=20 ttl=255 time=0.232 ms
64 bytes from 192.168.23.2: icmp_seq=21 ttl=255 time=0.232 ms
64 bytes from 192.168.23.2: icmp_seq=22 ttl=255 time=0.225 ms
64 bytes from 192.168.23.2: icmp_seq=23 ttl=255 time=0.223 ms
64 bytes from 192.168.23.2: icmp_seq=24 ttl=255 time=0.224 ms
64 bytes from 192.168.23.2: icmp_seq=25 ttl=255 time=469.175 ms
64 bytes from 192.168.23.2: icmp_seq=26 ttl=255 time=571.747 ms
64 bytes from 192.168.23.2: icmp_seq=27 ttl=255 time=0.253 ms
64 bytes from 192.168.23.2: icmp_seq=28 ttl=255 time=0.225 ms
64 bytes from 192.168.23.2: icmp_seq=29 ttl=255 time=0.229 ms
64 bytes from 192.168.23.2: icmp_seq=30 ttl=255 time=0.227 ms
64 bytes from 192.168.23.2: icmp_seq=31 ttl=255 time=1.222 ms
64 bytes from 192.168.23.2: icmp_seq=32 ttl=255 time=0.995 ms
64 bytes from 192.168.23.2: icmp_seq=33 ttl=255 time=0.238 ms
64 bytes from 192.168.23.2: icmp_seq=34 ttl=255 time=0.238 ms
64 bytes from 192.168.23.2: icmp_seq=35 ttl=255 time=0.230 ms
64 bytes from 192.168.23.2: icmp_seq=36 ttl=255 time=0.230 ms
64 bytes from 192.168.23.2: icmp_seq=37 ttl=255 time=679.469 ms
64 bytes from 192.168.23.2: icmp_seq=38 ttl=255 time=781.050 ms
64 bytes from 192.168.23.2: icmp_seq=39 ttl=255 time=0.221 ms
64 bytes from 192.168.23.2: icmp_seq=40 ttl=255 time=0.240 ms
^C
--- 192.168.23.2 ping statistics ---
41 packets transmitted, 41 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.221/81.489/781.050/195.848 ms
There is no switch involved in this pfsync connection, just a
single cable from NIC to NIC.
I see the same performance problem for incoming traffic on all
other network interfaces of redgatea and redgateb, MASTER and
BACKUP, even on the external connection. For outgoing traffic
(eg if I try to ping a 3rd host *from* redgateb) there is a
performance impact, too, but it is much lower:
redgateb# ping 10.100.100.101
PING 10.100.100.101 (10.100.100.101): 56 data bytes
64 bytes from 10.100.100.101: icmp_seq=0 ttl=64 time=0.291 ms
64 bytes from 10.100.100.101: icmp_seq=1 ttl=64 time=0.241 ms
64 bytes from 10.100.100.101: icmp_seq=2 ttl=64 time=0.235 ms
64 bytes from 10.100.100.101: icmp_seq=3 ttl=64 time=0.246 ms
64 bytes from 10.100.100.101: icmp_seq=4 ttl=64 time=1.176 ms
64 bytes from 10.100.100.101: icmp_seq=5 ttl=64 time=1.479 ms
64 bytes from 10.100.100.101: icmp_seq=6 ttl=64 time=0.220 ms
64 bytes from 10.100.100.101: icmp_seq=7 ttl=64 time=0.231 ms
64 bytes from 10.100.100.101: icmp_seq=8 ttl=64 time=0.228 ms
64 bytes from 10.100.100.101: icmp_seq=9 ttl=64 time=0.229 ms
64 bytes from 10.100.100.101: icmp_seq=10 ttl=64 time=0.242 ms
64 bytes from 10.100.100.101: icmp_seq=11 ttl=64 time=0.230 ms
64 bytes from 10.100.100.101: icmp_seq=12 ttl=64 time=0.244 ms
64 bytes from 10.100.100.101: icmp_seq=13 ttl=64 time=0.236 ms
64 bytes from 10.100.100.101: icmp_seq=14 ttl=64 time=0.236 ms
64 bytes from 10.100.100.101: icmp_seq=15 ttl=64 time=0.231 ms
64 bytes from 10.100.100.101: icmp_seq=16 ttl=64 time=1.465 ms
64 bytes from 10.100.100.101: icmp_seq=17 ttl=64 time=1.089 ms
64 bytes from 10.100.100.101: icmp_seq=18 ttl=64 time=0.220 ms
64 bytes from 10.100.100.101: icmp_seq=19 ttl=64 time=0.220 ms
64 bytes from 10.100.100.101: icmp_seq=20 ttl=64 time=0.233 ms
64 bytes from 10.100.100.101: icmp_seq=21 ttl=64 time=0.222 ms
^C
--- 10.100.100.101 ping statistics ---
22
Re: carp status master on both firewalls
--- Original Message --- On Friday, April 14th, 2023 at 7:14 AM, Janne Johansson wrote: > Not impossible to have switches(*) that dislike/filter/bug on > multicast too I guess, so I would suggest rigging the carps up (at > least temporary) with carppeer against the "real" ip of the remote > ext_if to make carp use normal unicast ip for sync and just see if it > helps. If it does, it is related to the boxes ability to talk > multicast and you would have to either stick with carppeer setup, or > "fix" the multicast issue, which can be hard to pin down where exactly > it is. Thank you Janne for pointing out the switch. I would have never thought about that. So indeed, I just upgraded my Cisco Catalyst 2960L switch the latest IOS version of 2022 and now the 2nd firewall correctly reports backup as status. I was running an IOS version from 2018. Strangely enough both firewalls are connected to that switch with both carp0 to the public VLAN and both carp1 to the private VLAN so I would have expected the same odd double master status on both carp interfaces and not just on carp0. But anyway it works now.
Re: carp status master on both firewalls
--- Original Message --- On Friday, April 14th, 2023 at 10:50 AM, Markus Wernig wrote: Thank you Markus for your answer, as mentioned to Janne it was the switch the problem. For the sake of documenting I answered your questions below. > - Do the two fw actually have a link on their carp0 carpdev interfaces? Yes. > If both are master, both should be sending out CARP advertisements, so > I'd try to run tcpdump on both external interfaces and look for those: > tcpdump -n -e -i carp0 proto carp I did that yesterday and for both firewalls I could see the CARPv2 advertisements. > - Did you enable CARP preemption? Try setting these via sysctl: > net.inet.carp.preempt=1 > net.inet.carp.log=3 I have CARP preemption enabled but my carp log level is 2 and not 3. > - In your config one fw has carpdev em2, the other carpdev em0. Could be > OK, or could be an error. Well spotted but indeed it is correct, both firewalls have different hardware and the first interface on the first firewall is em2 whereas on the 2nd firewall it is em0.
Re: carp status master on both firewalls
for my external carp interface both firewalls show master as status
The config is below for reference:
/etc/hostname.carp0 on fw1
inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em2 pass password
advskew 1
inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0
/etc/hostname.carp0 on fw2
inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em0 pass password advskew 128
inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0
On both firewalls I have added the following in /etc/pf.conf:
pass on { $ext_if $int_if } proto carp keep state (no-sync)
Did anyone already encounter this issue or has any idea what might be wrong?
Hard to tell without logs. Some things that come to mind:
- Do the two fw actually have a link on their carp0 carpdev interfaces?
If both are master, both should be sending out CARP advertisements, so
I'd try to run tcpdump on both external interfaces and look for those:
tcpdump -n -e -i carp0 proto carp
- Did you enable CARP preemption? Try setting these via sysctl:
net.inet.carp.preempt=1
net.inet.carp.log=3
- In your config one fw has carpdev em2, the other carpdev em0. Could be
OK, or could be an error.
carp status master on both firewalls
Hello,
I am running two OpenBSD 7.3 firewalls with pfsync and CARP for redundancy and
have one carp interface carp0 for the public internet and one carp interface
carp1 for my private (NAT) internal network. The private carp interface has
status master on the first firewall and status backup on the 2nd firewall so
all good here but for my external carp interface both firewalls show master as
status so I suspect I have something wrong here in my config or it is a bug.
The config is below for reference:
/etc/hostname.carp0 on fw1
inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em2 pass password
advskew 1
inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0
/etc/hostname.carp0 on fw2
inet x.x.x.114 255.255.255.240 x.x.x.127 vhid 40 carpdev em0 pass password
advskew 128
inet alias x.x.x.115 0xfff0
inet alias x.x.x.116 0xfff0
On both firewalls I have added the following in /etc/pf.conf:
pass on { $ext_if $int_if } proto carp keep state (no-sync)
Did anyone already encounter this issue or has any idea what might be wrong?
Best regards,
Mabi
Re: CARP and DHCP
On Sun, Jan 8, 2023 at 5:23 PM Nick Holland wrote: > > Does this actually maintain state? I'm thinking pfsync might > not work properly when the external interface "changes" like that. > It wouldn't actually matter much in *my case*, but I'm wondering > about the more general case. > > > I no expert, but I think so. I can be in a Teams meeting while I switch from master to slave, without loosing the connection.
Re: CARP and DHCP
On 1/6/23 02:31, Christer Solskogen wrote:
On Mon, Jan 2, 2023 at 5:14 PM Nick Holland
wrote:
hiya.
Goal: home (i.e., DHCP external network config) redundant
firewalls with CARP and PFSYNC.
Totally doable. I've been running it like that for the last 7 years at
home.
My ISP doesn't like it when the two firewalls have different mac-addresses,
same here. :)
so I have to do some spoofing on the slave machine.
ifstated is your very good friend here. My /etc/hostname.$extif is empty.
CARP is only in use for the internal interface.
This if my ifstated.conf on mster:
carp_up = "carp0.link.up"
carp_down = "!carp0.link.up"
carp_init = "carp0.link.unknown"
init-state auto
state auto {
if ($carp_up)
set-state fw_master
if !($carp_up)
set-state fw_slave
}
state fw_master {
init {
run "route -qn flush"
run "ifconfig em2 inet autoconf"
run "pfctl -f /etc/pf.conf"
}
if ($carp_down)
set-state fw_slave
if ($carp_init)
run "sleep 2"
}
state fw_slave {
init {
run "ifconfig em2 -inet"
run "route -qn flush"
run "route add default 192.168.0.3"
}
if ($carp_up)
set-state fw_master
}
Does this actually maintain state? I'm thinking pfsync might
not work properly when the external interface "changes" like that.
It wouldn't actually matter much in *my case*, but I'm wondering
about the more general case.
Thanks!
Nick.
Re: CARP and DHCP
On Mon, Jan 2, 2023 at 5:14 PM Nick Holland
wrote:
> hiya.
>
> Goal: home (i.e., DHCP external network config) redundant
> firewalls with CARP and PFSYNC.
>
>
Totally doable. I've been running it like that for the last 7 years at
home.
My ISP doesn't like it when the two firewalls have different mac-addresses,
so I have to do some spoofing on the slave machine.
ifstated is your very good friend here. My /etc/hostname.$extif is empty.
CARP is only in use for the internal interface.
This if my ifstated.conf on mster:
carp_up = "carp0.link.up"
carp_down = "!carp0.link.up"
carp_init = "carp0.link.unknown"
init-state auto
state auto {
if ($carp_up)
set-state fw_master
if !($carp_up)
set-state fw_slave
}
state fw_master {
init {
run "route -qn flush"
run "ifconfig em2 inet autoconf"
run "pfctl -f /etc/pf.conf"
}
if ($carp_down)
set-state fw_slave
if ($carp_init)
run "sleep 2"
}
state fw_slave {
init {
run "ifconfig em2 -inet"
run "route -qn flush"
run "route add default 192.168.0.3"
}
if ($carp_up)
set-state fw_master
}
CARP and DHCP
hiya. Goal: home (i.e., DHCP external network config) redundant firewalls with CARP and PFSYNC. Long ago, I think the word was "CARP and DHCP network configs don't work well together". A bit of searching man pages isn't showing me anything. A bit of googling is showing some old solutions that were fairly complicated. A lot has changed, lots of nifty new tools. Is there anything that would make a DHCP-configured redundant FW relatively straight-forward? I can think of a lot of reasons why this would NOT be an easy thing to accomplish, but maybe I've missed something. (Goal is to re-acquaint myself with CARP. I can accomplish that goal with a "buffer" machine between the CARP/PFSYNC FW and the outside Internet, but if I can skip the extra machine and get the benefits of redundancy, I'd like to do so). Nick.
Re: carp question
On Tue, Jun 28, 2022 at 2:58 PM Stuart Henderson wrote: > > So for this you would need to monitor the interface status and change > the default route, you couldn't rely on /etc/mygate. > > I don't. I use ifstated :-)
Re: carp question
On 2022-06-28, Christer Solskogen wrote: > On Tue, Jun 28, 2022 at 12:18 PM Łukasz Moskała wrote: > >> You wanted to set the CARP IP as default gateway on both master and >> backup, right? >> >> > No, the master is the gateway. So what would not make sense as all. > I would like to have the CARP IP as default gateway on the backup. (And > vice-versa, when the now-master becomes backup) So for this you would need to monitor the interface status and change the default route, you couldn't rely on /etc/mygate. But, if you do that anyway, you can just use the standard non-carp address on the backup, and the proper upstream router on the active machine
Re: carp question
On Tue, Jun 28, 2022 at 12:18 PM Łukasz Moskała wrote: > You wanted to set the CARP IP as default gateway on both master and > backup, right? > > No, the master is the gateway. So what would not make sense as all. I would like to have the CARP IP as default gateway on the backup. (And vice-versa, when the now-master becomes backup)
Re: carp question
Dnia Tue, Jun 28, 2022 at 11:36:55AM +0200, Christer Solskogen napisał(a): > On Tue, Jun 28, 2022 at 10:44 AM Stuart Henderson > wrote: > > > > > It makes no sense to set your own address as the default gateway? > > > > > It would *if* backup didn't respond to the carp IP. But it does, so no it > does not make sense. > The reason I was hoping it would work was the wording in the FAQ. You wanted to set the CARP IP as default gateway on both master and backup, right? Then, even if it would work as you thought, what would happen on master? You'd have CARP IP set as a gateway. The same CARP IP that is on this server. Equivalent of setting 127.0.0.1 as a gateway.
Re: carp question
On Tue, Jun 28, 2022 at 10:44 AM Stuart Henderson wrote: > > It makes no sense to set your own address as the default gateway? > > It would *if* backup didn't respond to the carp IP. But it does, so no it does not make sense. The reason I was hoping it would work was the wording in the FAQ.
Re: carp question
On 2022-06-28, Christer Solskogen wrote: > On Tue, Jun 28, 2022 at 10:21 AM Łukasz Moskała wrote: > >> >> What problem are you trying to solve? >> >> > Having identical config files on both the master and backup when it comes > to setting up the default gateway. I was hoping I could just use the carp > address as default gateway. It makes no sense to set your own address as the default gateway? -- Please keep replies on the mailing list.
Re: carp question
On Tue, Jun 28, 2022 at 10:21 AM Łukasz Moskała wrote: > > What problem are you trying to solve? > > Having identical config files on both the master and backup when it comes to setting up the default gateway. I was hoping I could just use the carp address as default gateway. -- chs
Re: carp question
Dnia Tue, Jun 28, 2022 at 10:03:25AM +0200, Christer Solskogen napisał(a): > On Tue, Jun 28, 2022 at 9:52 AM Łukasz Moskała wrote: > > > Run tcpdump on master, ping on backup. If you see pings in tcpdump, then > > master is responding. > > If not, backup is responding to itself. > > > > > Good catch. > The backup is responding to it self. But should it? In the FAQ I find this: > "The master host is the one that currently "holds" the shared IP; it > responds to any traffic or ARP requests directed towards that address." > So the question now is should master respond? > > -- > chs Master can't respond to that traffic, as it never goes over wire. Just like pinging 127.0.0.1 If you have 192.0.2.10/24 on em0, and ping 192.0.2.10 from host that has this IP, this traffic newer goes over network cable either. Also this allows you to bind service to that CARP IP on both hosts. If that address would "disappear" from interface when host is in backup mode, services running on backup host couldn't bind to that IP. For example you have CARP IP 192.0.2.200/32. You can bind httpd to 192.0.2.200:443 on both hosts, and have both of them running, but only master would receive traffic. Master is the only one who advertises this IP on the network. So from perspective of any other host, only master has this address. Maybe you can get userspace CARP implementation that removes IP from interface when in backup mode and adds it again when in master mode. This is how keepalived on linux works (but it uses VRRP, not CARP). What problem are you trying to solve? -- Łukasz Moskała
Re: carp question
On Tue, Jun 28, 2022 at 9:52 AM Łukasz Moskała wrote: > Run tcpdump on master, ping on backup. If you see pings in tcpdump, then > master is responding. > If not, backup is responding to itself. > > Good catch. The backup is responding to it self. But should it? In the FAQ I find this: "The master host is the one that currently "holds" the shared IP; it responds to any traffic or ARP requests directed towards that address." So the question now is should master respond? -- chs
Re: carp question
Dnia Tue, Jun 28, 2022 at 09:47:40AM +0200, Christer Solskogen napisał(a): > if you ping the carp ip from the backup, does the master respond or the > backup it self? > > -- > chs Hi, Run tcpdump on master, ping on backup. If you see pings in tcpdump, then master is responding. If not, backup is responding to itself. -- Łukasz Moskała
carp question
if you ping the carp ip from the backup, does the master respond or the backup it self? -- chs
Re: Re : iked + sasyncd + carp - doesn't take over
I have some more info (this time from physical machines): After a switchover I can see incoming flow on enc0 on the new master, and it IS decoded correctly. It is just not pushed out into the protected network. Additionally, the replay counters seem to be all in sync except for one - return tunnel to client on a backup node has replay counter inreased by 16384 (for example replay: rpl 167 on master and replay: rpl 16551 on backup). -- Paweł Kraszewski
Re: Re : iked + sasyncd + carp - doesn't take over
p curve25519 prf hmac-sha2-256) Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: msg 0x62f175f4f60 len 336 ref 1 to peer 10.0.1.162 Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: freeing msg 0x62f175f4f60 Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: msg 0x62f175bb7d0 len 272 ref 1 to peer 10.0.1.162 Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: freeing msg 0x62f175bb7d0 Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: msg 0x62f175f4330 len 272 ref 1 to peer 10.0.1.162 Mar 4 12:33:58 ipsec1 sasyncd[3474]: net_send_messages: freeing msg 0x62f175f4330 Connection seen from PASSIVE side: Mar 4 12:35:17 ipsec2 sasyncd[20476]: net_handle_messages: got msg type 1 len 280 from peer 10.0.1.161 Mar 4 12:35:17 ipsec2 sasyncd[20476]: pfkey_queue_message: pfkey ADD len 280 seq 18 Mar 4 12:35:17 ipsec2 sasyncd[20476]: net_handle_messages: got msg type 1 len 280 from peer 10.0.1.161 Mar 4 12:35:17 ipsec2 sasyncd[20476]: pfkey_queue_message: pfkey ADD len 280 seq 19 Mar 4 12:35:17 ipsec2 sasyncd[20476]: net_handle_messages: got msg type 1 len 224 from peer 10.0.1.161 Mar 4 12:35:17 ipsec2 sasyncd[20476]: pfkey_queue_message: pfkey X_ADDFLOW len 224 seq 20 Mar 4 12:35:17 ipsec2 sasyncd[20476]: net_handle_messages: got msg type 1 len 224 from peer 10.0.1.161 Mar 4 12:35:17 ipsec2 sasyncd[20476]: pfkey_queue_message: pfkey X_ADDFLOW len 224 seq 21 ipsecctl -v sa on BOTH sides (192.168.1.46 is my mobile WAN, 192.168.1.160 is CARP of ipsec, 10.1.0.190 is mobile tunnel IP, 10.0.0.0/24 is local LAN I'm granting access to): FLOWS: flow esp in from 10.1.0.190 to 10.0.0.0/24 peer 192.168.1.46 srcid FQDN/vpn.my.domain dstid FQDN/test type require flow esp out from 10.0.0.0/24 to 10.1.0.190 peer 192.168.1.46 srcid FQDN/vpn.my.domain dstid FQDN/test type require SAD: esp tunnel from 192.168.1.160 to 192.168.1.46 spi 0x2f538456 enc aes-128-gcm sa: spi 0x2f538456 auth gmac-aes-128 enc aes-gcm state mature replay 64 flags 0x204 lifetime_cur: alloc 0 bytes 0 add 1646393638 first 0 lifetime_hard: alloc 0 bytes 4294967296 add 10800 first 0 lifetime_soft: alloc 0 bytes 3809635991 add 9579 first 0 address_src: 192.168.1.160 address_dst: 192.168.1.46 identity_src: type fqdn id 0: FQDN/vpn.my.domain identity_dst: type fqdn id 0: FQDN/test udpencap: udpencap port 43052 tag: win7-FQDN/test counter: replay: rpl 1 esp tunnel from 192.168.1.46 to 192.168.1.160 spi 0xe56d3eef enc aes-128-gcm sa: spi 0xe56d3eef auth gmac-aes-128 enc aes-gcm state mature replay 64 flags 0x204 lifetime_cur: alloc 0 bytes 0 add 1646393638 first 0 lifetime_hard: alloc 0 bytes 4294967296 add 10800 first 0 lifetime_soft: alloc 0 bytes 4024384356 add 10119 first 0 address_src: 192.168.1.46 address_dst: 192.168.1.160 identity_src: type fqdn id 0: FQDN/test identity_dst: type fqdn id 0: FQDN/vpn.my.domain udpencap: udpencap port 43052 tag: win7-FQDN/test counter: replay: rpl 1 ikectl sh sa on ACTIVE only (PASSIVE is empty) iked_sas: 0xf64ca0257b0 rspi 0xbb3d2768049c9289 ispi 0x47c4ccf08d9d8699 192.168.1.160:4500->192.168.1.46:43052[10.1.0.190] ESTABLISHED r natt udpecap nexti 0x0 pol 0xf6453101000 sa_childsas: 0xf64ca022a00 ESP 0xe56d3eef in 192.168.1.46:43052 -> 192.168.1.160:4500 (LA) B=0x0 P=0xf64ca034200 @0xf64ca0257b0 sa_childsas: 0xf64ca034200 ESP 0x2f538456 out 192.168.1.160:4500 -> 192.168.1.46:43052 (L) B=0x0 P=0xf64ca022a00 @0xf64ca0257b0 sa_flows: 0xf64ca018400 ESP out 10.0.0.0/24 -> 10.1.0.190/32 [0]@-1 (L) @0xf64ca0257b0 sa_flows: 0xf64ca03a000 ESP in 10.1.0.190/32 -> 10.0.0.0/24 [0]@-1 (L) @0xf64ca0257b0 iked_activesas: 0xf64ca034200 ESP 0x2f538456 out 192.168.1.160:4500 -> 192.168.1.46:43052 (L) B=0x0 P=0xf64ca022a00 @0xf64ca0257b0 iked_activesas: 0xf64ca022a00 ESP 0xe56d3eef in 192.168.1.46:43052 -> 192.168.1.160:4500 (LA) B=0x0 P=0xf64ca034200 @0xf64ca0257b0 iked_flows: 0xf64ca03a000 ESP in 10.1.0.190/32 -> 10.0.0.0/24 [0]@-1 (L) @0xf64ca0257b0 iked_flows: 0xf64ca018400 ESP out 10.0.0.0/24 -> 10.1.0.190/32 [0]@-1 (L) @0xf64ca0257b0 iked_dstid_sas: 0xf64ca0257b0 rspi 0xbb3d2768049c9289 ispi 0x47c4ccf08d9d8699 192.168.1.160:4500->192.168.1.46:43052[10.1.0.190] ESTABLISHED r natt udpecap nexti 0x0 pol 0xf6453101000 Log on SLAVE after pausing MASTER: Mar 4 12:40:49 ipsec2 sasyncd[20476]: carp_update_state: switching state to MASTER Mar 4 12:40:49 ipsec2 sasyncd[20476]: net_ctl: sending my state MASTER to peer "10.0.1.161" Mar 4 12:40:49 ipsec2 sasyncd[20476]: net_send_messages: msg 0x6d895bbc3b0 len 64 ref 1 to peer 10.0.1.161
iked + sasyncd + carp - doesn't take over
Hello!
I'm trying to build a redundant IPSEC VPN concentrator.
What have I done by now:
* I have a working CARP. Verified from each side. 1-2 pings lost.
Works as expected.
* I have a working iked deployment. Test client can connect, sees
internal network as expected.
* I have a working pfsync. Pf states are replicated between nodes.
* I have a working sasyncd. Flows and SADs are replicated between nodes.
What doesn't work:
When the client is connected to a virtual CARP endpoint and I perform
a switchover, the new master doesn't pick up the communication.
NAT-t packages do come to a valid host, they are just not processed.
Iked compains with "ikev2_child_sa_acquire: flow wasn't found"
The full relevant configuration files follow:
Topology:
2 Identical Qemu's, OpenBSD 7.0, no conflicting MAC addresses
em0-s bridged together -> (WAN) -> strongswan on mobile phone
em1-s bridged together -> (LAN) -> IP to ping from mobile
em2-s bridged together -> (sync)
-
sysctl.conf
net.inet.carp.preempt=1
net.inet.ip.forwarding=1
-
hostname.carp0 (differences with | , hosts A|B)
inet 192.168.1.160 255.255.255.0 192.168.1.255 \
carpdev em0 \
group VPN \
pass passwd \
vhid 1 \
advskew 0|100
-
hostname.carp1
inet 10.0.0.254 255.255.255.0 10.0.0.255 \
carpdev em1 \
group VPN \
pass passwd \
vhid 2 \
advskew 0|100
-
hostname.em0
inet 192.168.1.161|162 255.255.255.0 NONE
-
hostname.em1
inet 10.0.0.161|162 255.255.255.0 NONE
-
hostname.em2
inet 10.0.1.161|162 255.255.255.0
-
hostname.enc0
inet 10.1.0.254 255.255.255.0
-
hostname.pfsync0
up \
syncdev em2 \
syncpeer 10.0.1.162|161
-
iked.conf
user "test" "password123"
set mobike
set enforcesingleikesa
set passive
ikev2 "VPN" esp \
from 10.0.0.0/24 to dynamic \
local 192.168.1.160 \
srcid vpn.my.domain \
eap "mschap-v2" \
config address 10.1.0.0/24 \
tag "$name-$id"
-
sasyncd.conf
peer 10.0.1.162|161
control iked
group VPN
interface carp0
listen on em2
sharedkey TAKEN_FROM_EXAMPLE
-
rc.conf.local
iked_flags=
ipsec=YES
sasyncd_flags=
ntpd_flags=NO
-
pf.conf
set skip on lo
pass quick on { em2 } proto pfsync keep state (no-sync)
pass on { em0 em1 } proto carp keep state (no-sync)
block return # block stateless traffic
pass # establish keep-state
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
pass in on em0 proto udp from any to (em0) port {isakmp, ipsec-nat-t}
tag IKED keep state
pass in on em0 proto esp from any to (em0) tag IKED keep state
pass in on em0 from (em0:network) to any
pass in on em1 from (em1:network) to any
pass in on em2 from (em2:network) to any
-
What do I miss?
Best regards,
--
Paweł Kraszewski
Re: CARP Cold Spare
On 9/24/21 6:13 PM, Don Tek wrote: Would there be any ‘problem’ with configuring a 2-machine CARP setup and then just keeping one machine powered-off until needed? I realize this defeats live failover, but this is not a requirement for my customer. I just want them to be able to, in the event of a primary machine failure, power-on the secondary and have it take over. Logic here is to otherwise not have the secondary sucking power off the UPS’s in the event of a power failure, or in general. Legit? Technically, should work fine. Administratively? you got a mess on your hands. With a fair amount of certainty, I can say you will run into at least the following administrative problems: * users added/changed to live box, not duplicated to spare. * Updates done to live box, not to spare * rule changes done to live box, not to spare. * Other settings changed on live box, not duplicated to spare. None of these are intrinsically CARP issues or solved by CARP at all but all of them are are going to be complicated by having a machine that is off when changes are made to the live one. All these issues have to be considered with a CARP setup, but with a machine powered off, you KNOW they won't be dealt with in a timely manner...which means they won't be there when you need them. If your goal is really to have one machine running, I'd suggest skipping CARP, and just mirror the drives on the primary and keep spare hw in reserve, and keep good backups of ALL configuration information. Drive fails? you got a mirror. HW fails? you got spare, move the drives, bring it up. Keep it simple, you will be happier. Nick.
Re: CARP Cold Spare
On 2021-09-25, leonard wrote: > What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and > 5 on surge protection. As long as your total draw is less than 1200 VA, for < > $200 canadian you have a cheap simple solution. Just put on on the ups side > and the other on the surge suppressor side. Or buy 2.leonard@on the road Then, when the UPS is depleted and auto shutdown either failed or wasn't setup in the first place, you can have two machines failing to start due to fsck failures, not just one! > > They experience multi-hour blackouts what seems like once a month; If they aren't spending the money on lower power servers, the chance of spending the money on a UPS and battery chain (or more likely, ATS and generator) capable of surviving multi-hour blackouts is slim. Seems to me the "cold spare" idea makes complete sense for the situation described. I would not bother with pfsync for this use. However, if the network config allows, I _would_ try to get some low power box (rpi or whatever) connected to serial console on both routers. -- Please keep replies on the mailing list.
Re: CARP Cold Spare
What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and 5 on surge protection. As long as your total draw is less than 1200 VA, for < $200 canadian you have a cheap simple solution. Just put on on the ups side and the other on the surge suppressor side. Or buy 2.leonard@on the road Original message From: Don Tek Date: 2021-09-25 11:40 (GMT-05:00) To: jslee Cc: misc@openbsd.org Subject: Re: CARP Cold Spare I'm not sure why the hardware matters, but the two machines are a couple HP 1U Gen 8 Xeon servers. Suffice to say, they are identical and have supported hardware configurations for OpenBSD.Of course I _could_ run one off direct power, but it would be a terrible idea. The location is notorious for power surges, blips that are enough to reboot servers and several-second brown-outs. So, not connected to the UPS is just asking for damages.They experience multi-hour blackouts what seems like once a month; this is where the desire to limit the draw on the UPS's comes from. To ensure we make it through without having to shut down.Remote access is of primary concern, both for me for support, since I'm geographically far enough away that being on-site is not feasible, and to the customer, who just wants to stay home and work on systems in the office.Configurations on the servers almost never change (simple firwall), so besides having to run a quick syspatch and reboot once at time of failover, I don't see maintenance being so bad. I keep config files backed-up otherwise centrally for quick restore to the running box as well.My primary concern here is if CARP / pfsync will have issues with the one machine being down a majority of the time. Based on the FAQ, I think not, but have no practical experience.> On Sep 25, 2021, at 3:00 AM, jslee wrote:> > Hi,> > You haven’t said anything about your hardware platform, but could you run one of them on non-UPS power? Then you’d still have one online when (*not* if) the UPS fails, and also they’ll both normally be online for maintenance, syspatch, config changes etc> > I do recall installing a pair of identical servers at the same time and having them both fail a year later within an hour of each other, both with seized CPU fans, so I am somewhat sympathetic to your idea. But I think the practical cost of maintenance may be rather high> > John> > >> On Sat, 25 Sep 2021, at 08:13, Don Tek wrote:>> Would there be any ‘problem’ with configuring a 2-machine CARP setup >> and then just keeping one machine powered-off until needed?>> >> I realize this defeats live failover, but this is not a requirement for >> my customer.>> >> I just want them to be able to, in the event of a primary machine >> failure, power-on the secondary and have it take over. Logic here is >> to otherwise not have the secondary sucking power off the UPS’s in the >> event of a power failure, or in general.>> >> Legit?
Re: CARP Cold Spare
Unsure what the power draw is on these guys yet, they just got them. They have redundant 450W Platinum power supplies. The "new" servers are completely overkill for the application, but this is a work-with-what's-available situation. They got these free from a friend and don't want to spend on new hardware, otherwise I'd just get them something "smaller" and supremely efficient. We already have redundant UPS's. With the 3 servers previously (main VM servers and 1 now-dead firewall) attached to battery-backed power, we were getting roughly 3.5 hours of runtime before UPS drop. Sometimes minutes can matter with these blackouts. I also want that with the one machine not running, it's not incurring any wear or tear just for the sake of hot failover. I've got no talent on site, so in the event of a hardware or file system failure, It's an extremely tedious experience for me to walk someone there through swapping hardware, wiring, or reinstalling an operating system. So, I'm looking for that turn-this-one-off-and-this-one-on recovery option, without the need to fiddle with any hardware. If this works the way I want, the only single point of failure will be their cable modem, and Comcast can handle replacing that. Option B is to just have identical configurations and have them need to swap the network wiring as part of the failover. Will still prevent the days of downtime we're incurring now due to this failure, but not as simple for them, if the CARP solution is solid. > On Sep 25, 2021, at 10:25 AM, leonard wrote: > > > What is the power draw? I use a 1500 VA apc backups with 6 outlets on ups and > 5 on surge protection. As long as your total draw is less than 1200 VA, for < > $200 canadian you have a cheap simple solution. Just put on on the ups side > and the other on the surge suppressor side. Or buy 2. > > > > leonard@on the road > > > Original message > From: Don Tek > Date: 2021-09-25 11:40 (GMT-05:00) > To: jslee > Cc: misc@openbsd.org > Subject: Re: CARP Cold Spare > > I'm not sure why the hardware matters, but the two machines are a couple HP > 1U Gen 8 Xeon servers. Suffice to say, they are identical and have supported > hardware configurations for OpenBSD. > > Of course I _could_ run one off direct power, but it would be a terrible > idea. The location is notorious for power surges, blips that are enough to > reboot servers and several-second brown-outs. So, not connected to the UPS > is just asking for damages. > > They experience multi-hour blackouts what seems like once a month; this is > where the desire to limit the draw on the UPS's comes from. To ensure we > make it through without having to shut down. > > Remote access is of primary concern, both for me for support, since I'm > geographically far enough away that being on-site is not feasible, and to the > customer, who just wants to stay home and work on systems in the office. > > Configurations on the servers almost never change (simple firwall), so > besides having to run a quick syspatch and reboot once at time of failover, I > don't see maintenance being so bad. I keep config files backed-up otherwise > centrally for quick restore to the running box as well. > > My primary concern here is if CARP / pfsync will have issues with the one > machine being down a majority of the time. Based on the FAQ, I think not, > but have no practical experience. > > > On Sep 25, 2021, at 3:00 AM, jslee wrote: > > > > Hi, > > > > You haven’t said anything about your hardware platform, but could you run > > one of them on non-UPS power? Then you’d still have one online when (*not* > > if) the UPS fails, and also they’ll both normally be online for > > maintenance, syspatch, config changes etc > > > > I do recall installing a pair of identical servers at the same time and > > having them both fail a year later within an hour of each other, both with > > seized CPU fans, so I am somewhat sympathetic to your idea. But I think the > > practical cost of maintenance may be rather high > > > > John > > > > > >> On Sat, 25 Sep 2021, at 08:13, Don Tek wrote: > >> Would there be any ‘problem’ with configuring a 2-machine CARP setup > >> and then just keeping one machine powered-off until needed? > >> > >> I realize this defeats live failover, but this is not a requirement for > >> my customer. > >> > >> I just want them to be able to, in the event of a primary machine > >> failure, power-on the secondary and have it take over. Logic here is > >> to otherwise not have the secondary sucking power off the UPS’s in the > >> event of a power failure, or in general. > >> > >> Legit? >
Re: CARP Cold Spare
I'm not sure why the hardware matters, but the two machines are a couple HP 1U Gen 8 Xeon servers. Suffice to say, they are identical and have supported hardware configurations for OpenBSD. Of course I _could_ run one off direct power, but it would be a terrible idea. The location is notorious for power surges, blips that are enough to reboot servers and several-second brown-outs. So, not connected to the UPS is just asking for damages. They experience multi-hour blackouts what seems like once a month; this is where the desire to limit the draw on the UPS's comes from. To ensure we make it through without having to shut down. Remote access is of primary concern, both for me for support, since I'm geographically far enough away that being on-site is not feasible, and to the customer, who just wants to stay home and work on systems in the office. Configurations on the servers almost never change (simple firwall), so besides having to run a quick syspatch and reboot once at time of failover, I don't see maintenance being so bad. I keep config files backed-up otherwise centrally for quick restore to the running box as well. My primary concern here is if CARP / pfsync will have issues with the one machine being down a majority of the time. Based on the FAQ, I think not, but have no practical experience. > On Sep 25, 2021, at 3:00 AM, jslee wrote: > > Hi, > > You haven’t said anything about your hardware platform, but could you run one > of them on non-UPS power? Then you’d still have one online when (*not* if) > the UPS fails, and also they’ll both normally be online for maintenance, > syspatch, config changes etc > > I do recall installing a pair of identical servers at the same time and > having them both fail a year later within an hour of each other, both with > seized CPU fans, so I am somewhat sympathetic to your idea. But I think the > practical cost of maintenance may be rather high > > John > > >> On Sat, 25 Sep 2021, at 08:13, Don Tek wrote: >> Would there be any ‘problem’ with configuring a 2-machine CARP setup >> and then just keeping one machine powered-off until needed? >> >> I realize this defeats live failover, but this is not a requirement for >> my customer. >> >> I just want them to be able to, in the event of a primary machine >> failure, power-on the secondary and have it take over. Logic here is >> to otherwise not have the secondary sucking power off the UPS’s in the >> event of a power failure, or in general. >> >> Legit?
Re: CARP Cold Spare
Hi, You haven’t said anything about your hardware platform, but could you run one of them on non-UPS power? Then you’d still have one online when (*not* if) the UPS fails, and also they’ll both normally be online for maintenance, syspatch, config changes etc I do recall installing a pair of identical servers at the same time and having them both fail a year later within an hour of each other, both with seized CPU fans, so I am somewhat sympathetic to your idea. But I think the practical cost of maintenance may be rather high John On Sat, 25 Sep 2021, at 08:13, Don Tek wrote: > Would there be any ‘problem’ with configuring a 2-machine CARP setup > and then just keeping one machine powered-off until needed? > > I realize this defeats live failover, but this is not a requirement for > my customer. > > I just want them to be able to, in the event of a primary machine > failure, power-on the secondary and have it take over. Logic here is > to otherwise not have the secondary sucking power off the UPS’s in the > event of a power failure, or in general. > > Legit?
Re: CARP Cold Spare
> On Sep 24, 2021, at 6:16 PM, Don Tek wrote: > > Would there be any ‘problem’ with configuring a 2-machine CARP setup and > then just keeping one machine powered-off until needed? > > I realize this defeats live failover, but this is not a requirement for my > customer. > > I just want them to be able to, in the event of a primary machine failure, > power-on the secondary and have it take over. Logic here is to otherwise not > have the secondary sucking power off the UPS’s in the event of a power > failure, or in general. > > Legit? > Sounds legit to me. Let’s you share the IP safely and easily, up or down.
CARP Cold Spare
Would there be any ‘problem’ with configuring a 2-machine CARP setup and then just keeping one machine powered-off until needed? I realize this defeats live failover, but this is not a requirement for my customer. I just want them to be able to, in the event of a primary machine failure, power-on the secondary and have it take over. Logic here is to otherwise not have the secondary sucking power off the UPS’s in the event of a power failure, or in general. Legit?
Re: [EXTERNAL] Why demotion counter for group carp is set to 33 on boot?
- Original Message - > From: "Tom K" > To: "misc" > Sent: Tuesday, July 13, 2021 3:32:04 AM > Subject: [EXTERNAL] Why demotion counter for group carp is set to 33 on boot? > Hallo, > > why demotion counter for group carp is set to 33 on boot? This is the > primary firewall and there are no adskew settings in all hostname.carpX > files or anywhere else. > Because of this the other firewall which should be normaly the standby > (adskew 100), is always MASTER (comes up with carp demote count 0). > > > > I can do "/sbin/ifconfig -g carp -carpdemote 33" in rc.local then this > system takes the MASTER rule and work together with the 2nd system as > expected. > This ar physical machines. I try to simulate this on vmware, but there > is everything fine. Both system starting with demote count 0. > > > I would appreciate any hint to understand this. > > Tom Greetings Tom: I don't have an answer for you, but I see the same behavior across a number of different hardware platforms (all amd64) and across all recent versions of OpenBSD (6.3+ for sure - maybe further back). I have pairs of machines as firewalls at remote sites. The only time that I reboot them is for patching and they sometimes get rebooted in an unplanned fashion due to power loss. It does not happen every time the systems are rebooted - maybe 1 time in 20. It happens often enough that checking the carp demotion counters after reboot is now part of the standard patching procedure and our monitoring system looks for and fixes the situation. It's always the box we consider 'primary' (advskew 10 vs. the secondary with advskew 100), and the carpdemote value is always set to 33. I can't be 100% certain, but I don't think I've ever seen it happen with the unplanned reboot/power loss. It only ever seems to happen after a syspatch and reboot. I have carp.preempt enabled, and I have suspected that the problem lies in there somewhere, but I have no evidence and it hasn't been enough of a problem to justify digging into it. Sorry I don't have an answer for you. I just wanted you to know that it isn't just you. -Scott
Re: Why demotion counter for group carp is set to 33 on boot?
On 7/13/21 9:32 AM, Tom K wrote: > why demotion counter for group carp is set to 33 on boot? This is the > primary firewall and there are no adskew settings in all hostname.carpX > files or anywhere else. > Because of this the other firewall which should be normaly the standby > (adskew 100), is always MASTER (comes up with carp demote count 0). I remember similar symptoms when some of my vlan interfaces were blocking carp traffic. I think I had to add an according rule for every interface like this: pass quick on vlan230 inet proto carp from any to any keep state (no-sync) In another case one of the interfaces on the master was misconfigured (some typo in hostname.if). Maybe setting net.inet.carp.log=3 also shows more info. best /m
Re: Why demotion counter for group carp is set to 33 on boot?
Am 2021-07-14 13:01, schrieb Stefan Sperling: If the demote count never drops then perhaps pfsync traffic isn't passing properly? tcpdump on pfsync device shows me PFSYNCv6 traffic all the time as well comparing the results of "pfctl -s state" on both systems shows no differences. The same setup in vmware simulation works as expected. The differences to the live environment, are installed both systems on different physical machines and are between some switches.
Re: Why demotion counter for group carp is set to 33 on boot?
On Wed, Jul 14, 2021 at 11:38:15AM +0200, Tom K wrote: > > > > but why? If I reboot the other node, the system become MASTER. > > > > That is because the other system stops sending carp announcements > > when you reboot it. This is unrelated to the demote counter. The demote > > counter only matters as long as another carp MASTER remains visible. > > A forced failover like this could break active connections. > > > > The pfsync interfaces adds 32 to the carp demote counter when it comes > > up, > > and it removes 32 from the demote counter once it has obtained an > > up-to-date > > copy of the state table, which can take some time. > > > > This prevents the box from becoming MASTER while it may not yet know > > about all the currently active connections. > > > > But why the 1st system is switchback to BACKUP if the 2nd system is Up > again? Normaly 1st should stay MASTER, because at this time, the 1st one > have the most recent state table which should be now send to the 2nd one. > Yes, it's because of the higher demotioncount then the 2nd system, but if > the 1st one standalone, it should self demoted to 0/1 bei pfsync because > there is no other system? > > I wait more then an hour, but the system is still on 33. So it seems the > state table is never synced completly, but if I compare both with "pfctl -s > state" they look are in sync - strange. > > I never had this issue and we use a lot of cluster setups like these in the > past. Yes that doesn't seem right. If you have the net.inet.carp.preempt sysctl set then the machine with a lower adskew value should move into BACKUP, provided the demote count is equal. But if the demote count is not equal then of course the machine with a higher demote count will remain in BACKUP state. If the demote count never drops then perhaps pfsync traffic isn't passing properly?
Re: Why demotion counter for group carp is set to 33 on boot?
but why? If I reboot the other node, the system become MASTER. That is because the other system stops sending carp announcements when you reboot it. This is unrelated to the demote counter. The demote counter only matters as long as another carp MASTER remains visible. A forced failover like this could break active connections. The pfsync interfaces adds 32 to the carp demote counter when it comes up, and it removes 32 from the demote counter once it has obtained an up-to-date copy of the state table, which can take some time. This prevents the box from becoming MASTER while it may not yet know about all the currently active connections. But why the 1st system is switchback to BACKUP if the 2nd system is Up again? Normaly 1st should stay MASTER, because at this time, the 1st one have the most recent state table which should be now send to the 2nd one. Yes, it's because of the higher demotioncount then the 2nd system, but if the 1st one standalone, it should self demoted to 0/1 bei pfsync because there is no other system? I wait more then an hour, but the system is still on 33. So it seems the state table is never synced completly, but if I compare both with "pfctl -s state" they look are in sync - strange. I never had this issue and we use a lot of cluster setups like these in the past.
Re: Why demotion counter for group carp is set to 33 on boot?
On Wed, Jul 14, 2021 at 08:28:09AM +0200, Tom K wrote: > JP, > > but why? If I reboot the other node, the system become MASTER. That is because the other system stops sending carp announcements when you reboot it. This is unrelated to the demote counter. The demote counter only matters as long as another carp MASTER remains visible. A forced failover like this could break active connections. > At this time, > no other system has more recent rule states and the 2nd system comes back, > the 1st will fall to BACKUP although it has no adskew settings and > demotecount still on 33. > I'm very confused about this issue. The pfsync interfaces adds 32 to the carp demote counter when it comes up, and it removes 32 from the demote counter once it has obtained an up-to-date copy of the state table, which can take some time. This prevents the box from becoming MASTER while it may not yet know about all the currently active connections. Just don't worry about it. It is supposed to happen this way.
Re: Why demotion counter for group carp is set to 33 on boot?
JP, but why? If I reboot the other node, the system become MASTER. At this time, no other system has more recent rule states and the 2nd system comes back, the 1st will fall to BACKUP although it has no adskew settings and demotecount still on 33. I'm very confused about this issue. Am 2021-07-13 18:12, schrieb Jorge Peixoto: Tom, Assuming the fw cluster is properly set up, I guess because PF rule states is unsync. As times goes by, states gets synchronized. JP
Re: Why demotion counter for group carp is set to 33 on boot?
Tom, Assuming the fw cluster is properly set up, I guess because PF rule states is unsync. As times goes by, states gets synchronized. JP Em ter., 13 de jul. de 2021 às 05:50, Tom K escreveu: > Hallo, > > why demotion counter for group carp is set to 33 on boot? This is the > primary firewall and there are no adskew settings in all hostname.carpX > files or anywhere else. > Because of this the other firewall which should be normaly the standby > (adskew 100), is always MASTER (comes up with carp demote count 0). > > OpenBSD foobar 6.7 GENERIC.MP#182 amd64 > > Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group carp by 32 to > 160 (pfsync init) > Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group pfsync by 32 > to 32 (pfsync init) > Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group carp by 1 to > 161 (pfsync bulk start) > Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group pfsync by 1 to > 33 (pfsync bulk start) > Jul 11 20:43:19 foobar /bsd: carp: carp0 demoted group carp by 1 to 162 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp1 demoted group carp by 1 to 163 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp10 demoted group carp by 1 to > 164 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp11 demoted group carp by 1 to > 165 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp12 demoted group carp by 1 to > 166 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp13 demoted group carp by 1 to > 167 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp14 demoted group carp by 1 to > 168 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp15 demoted group carp by 1 to > 169 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp16 demoted group carp by 1 to > 170 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp17 demoted group carp by 1 to > 171 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp2 demoted group carp by 1 to 172 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp3 demoted group carp by 1 to 173 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp5 demoted group carp by 1 to 174 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp6 demoted group carp by 1 to 175 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp7 demoted group carp by 1 to 176 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp8 demoted group carp by 1 to 177 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp: carp9 demoted group carp by 1 to 178 > (carpdev) > Jul 11 20:43:19 foobar /bsd: carp15: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp15 demoted group carp by -1 to > 177 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp0: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp0 demoted group carp by -1 to > 176 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp12: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp12 demoted group carp by -1 to > 175 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp3: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp3 demoted group carp by -1 to > 174 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp13: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp13 demoted group carp by -1 to > 173 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp2: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp2 demoted group carp by -1 to > 172 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp16: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp16 demoted group carp by -1 to > 171 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp9: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp9 demoted group carp by -1 to > 170 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp8: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp8 demoted group carp by -1 to > 169 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp14: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp14 demoted group carp by -1 to > 168 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp11: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp11 demoted group carp by -1 to > 167 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp10: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp10 demoted group carp by -1 to > 166 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp1: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp1 demoted group carp by -1 to > 165 (carpdev) > Jul 11 20:43:19 foobar /bsd: carp7: state transition: INIT -> BACKUP > Jul 11 20:43:19 foobar /bsd: carp: carp7 demoted group car
Why demotion counter for group carp is set to 33 on boot?
Hallo, why demotion counter for group carp is set to 33 on boot? This is the primary firewall and there are no adskew settings in all hostname.carpX files or anywhere else. Because of this the other firewall which should be normaly the standby (adskew 100), is always MASTER (comes up with carp demote count 0). OpenBSD foobar 6.7 GENERIC.MP#182 amd64 Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group carp by 32 to 160 (pfsync init) Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init) Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group carp by 1 to 161 (pfsync bulk start) Jul 11 20:43:19 foobar /bsd: carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start) Jul 11 20:43:19 foobar /bsd: carp: carp0 demoted group carp by 1 to 162 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp1 demoted group carp by 1 to 163 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp10 demoted group carp by 1 to 164 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp11 demoted group carp by 1 to 165 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp12 demoted group carp by 1 to 166 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp13 demoted group carp by 1 to 167 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp14 demoted group carp by 1 to 168 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp15 demoted group carp by 1 to 169 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp16 demoted group carp by 1 to 170 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp17 demoted group carp by 1 to 171 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp2 demoted group carp by 1 to 172 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp3 demoted group carp by 1 to 173 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp5 demoted group carp by 1 to 174 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp6 demoted group carp by 1 to 175 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp7 demoted group carp by 1 to 176 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp8 demoted group carp by 1 to 177 (carpdev) Jul 11 20:43:19 foobar /bsd: carp: carp9 demoted group carp by 1 to 178 (carpdev) Jul 11 20:43:19 foobar /bsd: carp15: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp15 demoted group carp by -1 to 177 (carpdev) Jul 11 20:43:19 foobar /bsd: carp0: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp0 demoted group carp by -1 to 176 (carpdev) Jul 11 20:43:19 foobar /bsd: carp12: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp12 demoted group carp by -1 to 175 (carpdev) Jul 11 20:43:19 foobar /bsd: carp3: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp3 demoted group carp by -1 to 174 (carpdev) Jul 11 20:43:19 foobar /bsd: carp13: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp13 demoted group carp by -1 to 173 (carpdev) Jul 11 20:43:19 foobar /bsd: carp2: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp2 demoted group carp by -1 to 172 (carpdev) Jul 11 20:43:19 foobar /bsd: carp16: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp16 demoted group carp by -1 to 171 (carpdev) Jul 11 20:43:19 foobar /bsd: carp9: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp9 demoted group carp by -1 to 170 (carpdev) Jul 11 20:43:19 foobar /bsd: carp8: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp8 demoted group carp by -1 to 169 (carpdev) Jul 11 20:43:19 foobar /bsd: carp14: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp14 demoted group carp by -1 to 168 (carpdev) Jul 11 20:43:19 foobar /bsd: carp11: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp11 demoted group carp by -1 to 167 (carpdev) Jul 11 20:43:19 foobar /bsd: carp10: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp10 demoted group carp by -1 to 166 (carpdev) Jul 11 20:43:19 foobar /bsd: carp1: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp1 demoted group carp by -1 to 165 (carpdev) Jul 11 20:43:19 foobar /bsd: carp7: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp7 demoted group carp by -1 to 164 (carpdev) Jul 11 20:43:19 foobar /bsd: carp6: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp6 demoted group carp by -1 to 163 (carpdev) Jul 11 20:43:19 foobar /bsd: carp5: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp5 demoted group carp by -1 to 162 (carpdev) Jul 11 20:43:19 foobar /bsd: carp17: state transition: INIT -> BACKUP Jul 11 20:43:19 foobar /bsd: carp: carp17 demoted group carp by -1 to 161 (carpdev) Jul 11 20:43:19 foobar /bsd: carp0: state transition: BACKUP -> MASTER Jul 11 20:43:19 foobar /bsd: c
Re: carp backup and disconnecting ssh session
MJ J(mikedotjack...@gmail.com) on 2021.05.23 17:58:47 +0300: > Hi, > > I have a carp master and backup on a pair of one-armed Rapsberry Pi 4B > devices (router1 and router2) and when I ssh to the backup using the > carp IP as my gateway, it repeatedly throws me out after a few seconds > with the message: > > My laptop's network config: > --- > IP: 192.168.4.109 > Subnet mask: 255.255.255.0 > Gateway: 192.168.4.1 > > Both RPI4s are connected to switchports with packets tagged for VLANs > 2,3,4,6 and the network devices don't have IP configuration - > everything is configured on VLAN interfaces with the single parent > interface bse0. CARP failover actually works as expected, but as > mentioned I am unable to maintain an ssh session with the backup > "router2" while using the carp IPs as my network gateway. > > Network switch is a Zyxel GS1200-8 with firmware V2.00(ABME.0)C0. Loop > prevention is enabled and I have also tested with it disabled to no > avail. > > What happens: > --- > $ ssh 10.0.1.101 > Last login: Sun May 23 17:44:21 2021 from 10.0.1.100 > OpenBSD 6.9 (GENERIC.MP) #1134: Sun Apr 18 01:53:35 MDT 2021 > router2# > router2# client_loop: send disconnect: Broken pipe you ssh from 192.168.4.109 to 10.0.1.101? My best guess is that you have asymetric routing and your carp master router1 only sees one direction of the traffic: laptop -> router1 -> router2 and router2 -> laptop because router2 has your laptop network locally on vlan6. Solution: ssh to 192.168.4.3. > > > Router 1 network config: > --- > router1# cat hostname.bse0 > up > > router1# cat hostname.vlan2 > 172.16.1.6/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC > description "private segment with router2" > > router1# cat hostname.vlan3 > 10.0.1.100/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL > description "router1 internal interface" > > router1# cat hostname.vlan4 > 192.168.1.252/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT > description "unmigrated shit" > > router1# cat hostname.vlan6 > 192.168.4.2/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI > description "Time-Capsule Wifi" > > router1# cat hostname.carp4 > 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 1 > description "TC-WIFI gateway" > > router1# cat hostname.carp6 > 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 1 > description "TC-WIFI gateway" > > > Router2 network config: > --- > router2# cat hostname.bse0 > up > > router2# cat hostname.vlan2 > 172.16.1.7/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC > description "private segment with router1" > > router2# cat hostname.vlan3 > 10.0.1.101/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL > description "router2 internal interface" > > router2# cat hostname.vlan4 > 192.168.1.253/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT > description "unmigrated shit" > > router2# cat hostname.vlan6 > 192.168.4.3/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI > description "Time-Capsule Wifi" > > router2# cat hostname.carp4 > 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 128 > description "TC-WIFI gateway" > > router2# cat hostname.carp6 > 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 128 > description "TC-WIFI gateway" > > > Any tips much appreciated. > > -mike > --
carp backup and disconnecting ssh session
Hi, I have a carp master and backup on a pair of one-armed Rapsberry Pi 4B devices (router1 and router2) and when I ssh to the backup using the carp IP as my gateway, it repeatedly throws me out after a few seconds with the message: My laptop's network config: --- IP: 192.168.4.109 Subnet mask: 255.255.255.0 Gateway: 192.168.4.1 Both RPI4s are connected to switchports with packets tagged for VLANs 2,3,4,6 and the network devices don't have IP configuration - everything is configured on VLAN interfaces with the single parent interface bse0. CARP failover actually works as expected, but as mentioned I am unable to maintain an ssh session with the backup "router2" while using the carp IPs as my network gateway. Network switch is a Zyxel GS1200-8 with firmware V2.00(ABME.0)C0. Loop prevention is enabled and I have also tested with it disabled to no avail. What happens: --- $ ssh 10.0.1.101 Last login: Sun May 23 17:44:21 2021 from 10.0.1.100 OpenBSD 6.9 (GENERIC.MP) #1134: Sun Apr 18 01:53:35 MDT 2021 router2# router2# client_loop: send disconnect: Broken pipe Router 1 network config: --- router1# cat hostname.bse0 up router1# cat hostname.vlan2 172.16.1.6/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC description "private segment with router2" router1# cat hostname.vlan3 10.0.1.100/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL description "router1 internal interface" router1# cat hostname.vlan4 192.168.1.252/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT description "unmigrated shit" router1# cat hostname.vlan6 192.168.4.2/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI description "Time-Capsule Wifi" router1# cat hostname.carp4 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 1 description "TC-WIFI gateway" router1# cat hostname.carp6 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 1 description "TC-WIFI gateway" Router2 network config: --- router2# cat hostname.bse0 up router2# cat hostname.vlan2 172.16.1.7/24 172.16.1.255 parent bse0 vnetid 2 group PFSYNC description "private segment with router1" router2# cat hostname.vlan3 10.0.1.101/24 10.0.1.255 parent bse0 vnetid 3 group INTERNAL description "router2 internal interface" router2# cat hostname.vlan4 192.168.1.253/24 192.168.1.255 parent bse0 vnetid 4 group OLDSHIT description "unmigrated shit" router2# cat hostname.vlan6 192.168.4.3/24 192.168.4.255 parent bse0 vnetid 6 group TCWIFI description "Time-Capsule Wifi" router2# cat hostname.carp4 192.168.1.1/24 carpdev vlan4 pass fukdissh1t vhid 41 advskew 128 description "TC-WIFI gateway" router2# cat hostname.carp6 192.168.4.1/24 carpdev vlan6 pass fukdissh1t vhid 61 advskew 128 description "TC-WIFI gateway" Any tips much appreciated. -mike
Re: Technical Documentation - CARP
Hi Jannick the man pages are also a good up to date source of information... sometimes a paper from a few years ago states something like X/Y is not supported... but as an OpenBSD developer once quiped "yes we do add features from time to time" :) so the papers can give really good context and insights... but refer to the manuals also to validate any improved syntax and or features Hope this helps Tom Smyth On Tue, 13 Apr 2021 at 09:34, jannick Weiss wrote: > > Hello,my name is Jannick Weiss and i am currently in the process of taking > my education as a datatechnician. As part of my education i have to do a > presentation on a self-elected subject and i have chosen to talk about CARP. > > It is my understanding that it is you (OpenBSD) that have developed CARP. > I am having trouble finding information about CARP, such as the different > states the protocol goes through or how the election of the master node > works specifically. > If you can provide any documentation on CARP it would be greatly > appreciated. > > In advance, thank you for any help you may provide. > > Best regards > > Jannick Weiss -- Kindest regards, Tom Smyth.
Re: Technical Documentation - CARP
On 2021-04-13, Janne Johansson wrote: > Den tis 13 apr. 2021 kl 10:29 skrev jannick Weiss : >> Hello,my name is Jannick Weiss and i am currently in the process of taking >> my education as a datatechnician. As part of my education i have to do a >> presentation on a self-elected subject and i have chosen to talk about CARP. >> >> It is my understanding that it is you (OpenBSD) that have developed CARP. >> I am having trouble finding information about CARP, such as the different >> states the protocol goes through or how the election of the master node >> works specifically. >> If you can provide any documentation on CARP it would be greatly >> appreciated. > > https://www.openbsd.org/events.html lists a few talks some 15 years > ago which focused on PF and Carp, those might help. https://github.com/jedisct1/UCarp has some useful information. There's no formal documentation for the protocol afaik. > Googling "openbsd carp design" turned this PDF up, > https://core.ac.uk/download/pdf/17210042.pdf from 2006 which perhaps > dives a bit deeper. It's a bit wrong though, I noticed it says "encrypted" - it's authenticated but not encrypted. Doesn't go much into the protocol details either.
Re: Technical Documentation - CARP
Den tis 13 apr. 2021 kl 10:29 skrev jannick Weiss : > Hello,my name is Jannick Weiss and i am currently in the process of taking > my education as a datatechnician. As part of my education i have to do a > presentation on a self-elected subject and i have chosen to talk about CARP. > > It is my understanding that it is you (OpenBSD) that have developed CARP. > I am having trouble finding information about CARP, such as the different > states the protocol goes through or how the election of the master node > works specifically. > If you can provide any documentation on CARP it would be greatly > appreciated. https://www.openbsd.org/events.html lists a few talks some 15 years ago which focused on PF and Carp, those might help. Googling "openbsd carp design" turned this PDF up, https://core.ac.uk/download/pdf/17210042.pdf from 2006 which perhaps dives a bit deeper. -- May the most significant bit of your life be positive.
Technical Documentation - CARP
Hello,my name is Jannick Weiss and i am currently in the process of taking my education as a datatechnician. As part of my education i have to do a presentation on a self-elected subject and i have chosen to talk about CARP. It is my understanding that it is you (OpenBSD) that have developed CARP. I am having trouble finding information about CARP, such as the different states the protocol goes through or how the election of the master node works specifically. If you can provide any documentation on CARP it would be greatly appreciated. In advance, thank you for any help you may provide. Best regards Jannick Weiss
Re: No advertisements from CARP master
On 2021-02-20, Dev Op wrote: > Hello, collegues! > > In vlan2 I have 4 routers: rt1 (master) and rt2 (slave) grouped into VHID > 50 in terms of CARP; rt3 (master) and rt4 (slave) grouped into VHID 2. Why > don't I see carp advertisements from rt1? Instead, I see carp announcements > only from rt3 (vhid2). Where am I wrong? > > rt1: > > -> % ifconfig carp2 | grep -v inet > carp2: flags=8843 mtu 1500 > lladdr 00:00:5e:00:01:32 > description: Interlink > index 5 priority 15 llprio 3 > carp: MASTER carpdev vlan2 vhid 50 advbase 1 advskew 0 > groups: carp > status: master > ks1@rt1 [05:44:47] [~] > -> % sudo tcpdump -c 10 -ni carp2 proto carp > tcpdump: listening on carp2, link-type EN10MB > 05:44:54.003157 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:44:55.003217 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:44:56.003236 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:44:57.003276 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:44:58.003313 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:44:59.003354 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:45:00.003398 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:45:01.003431 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:45:02.003475 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > 05:45:03.003512 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0 > (DF) [tos 0x10] > ks1@rt1 [05:45:03] [~] > -> % sysctl net.inet.carp > net.inet.carp.allow=1 > net.inet.carp.preempt=1 > net.inet.carp.log=2 > ks1@rt1 [05:46:40] [~] > -> % uname -r > 6.8 > ks1@rt1 [05:57:11] [~] > -> % sudo sysctl net.inet.carp.log=7 > net.inet.carp.log: 2 -> 7 > ks1@rt1 [05:57:25] [~] > -> % tail -f /var/log/messages > Feb 20 05:57:57 border1 /bsd: carp2: ip_output failed: 13 > Feb 20 05:57:58 border1 /bsd: carp2: ip_output failed: 13 > Feb 20 05:57:59 border1 /bsd: carp2: ip_output failed: 13 > Feb 20 05:58:00 border1 /bsd: carp2: ip_output failed: 13 > Feb 20 05:58:01 border1 /bsd: carp2: ip_output failed: 13 > ... $ grep 13 /usr/include/sys/errno.h #define EACCES 13 /* Permission denied */ Check that your PF rules allow sending carp's multicast packets. > I don't know what it means. Do I understand correctly that there can be > only one master in the multicast domain that can send carp advertisements? > Thanks. All carp hosts send advetisements, whether they're primary or backup.
No advertisements from CARP master
Hello, collegues!
In vlan2 I have 4 routers: rt1 (master) and rt2 (slave) grouped into VHID
50 in terms of CARP; rt3 (master) and rt4 (slave) grouped into VHID 2. Why
don't I see carp advertisements from rt1? Instead, I see carp announcements
only from rt3 (vhid2). Where am I wrong?
rt1:
-> % ifconfig carp2 | grep -v inet
carp2: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:32
description: Interlink
index 5 priority 15 llprio 3
carp: MASTER carpdev vlan2 vhid 50 advbase 1 advskew 0
groups: carp
status: master
ks1@rt1 [05:44:47] [~]
-> % sudo tcpdump -c 10 -ni carp2 proto carp
tcpdump: listening on carp2, link-type EN10MB
05:44:54.003157 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:44:55.003217 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:44:56.003236 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:44:57.003276 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:44:58.003313 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:44:59.003354 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:45:00.003398 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:45:01.003431 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:45:02.003475 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
05:45:03.003512 CARPv2-advertise 36: vhid=2 advbase=1 advskew=0 demote=0
(DF) [tos 0x10]
ks1@rt1 [05:45:03] [~]
-> % sysctl net.inet.carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2
ks1@rt1 [05:46:40] [~]
-> % uname -r
6.8
ks1@rt1 [05:57:11] [~]
-> % sudo sysctl net.inet.carp.log=7
net.inet.carp.log: 2 -> 7
ks1@rt1 [05:57:25] [~]
-> % tail -f /var/log/messages
Feb 20 05:57:57 border1 /bsd: carp2: ip_output failed: 13
Feb 20 05:57:58 border1 /bsd: carp2: ip_output failed: 13
Feb 20 05:57:59 border1 /bsd: carp2: ip_output failed: 13
Feb 20 05:58:00 border1 /bsd: carp2: ip_output failed: 13
Feb 20 05:58:01 border1 /bsd: carp2: ip_output failed: 13
...
I don't know what it means. Do I understand correctly that there can be
only one master in the multicast domain that can send carp advertisements?
Thanks.
Regards,
Den
Re: seeing carp interface state change for unknown reason ; cluestick hunting
On 2/7/21 1:38 AM, Bryan Stenson wrote: 31 RTM_IFINFO: iface status change: len 168, if# 3, name cnmac2, link: no carrier, mtu: 1500, Just grasping for something here...my next steps are to swap this unit out with the other one (to try and eliminate hardware failure of THIS unit). Any other suggestions? Check the switch interface for any errors and messages.
Re: seeing carp interface state change for unknown reason ; cluestick hunting
Thanks for the response. I've mounted a ramdisk at /mnt and have run "doas route -n monitor > /mnt/route.monitor" in a tmux session for a few days. Here are some details: erl3-01$ grep carp1 route.monitor | sort | uniq -c 91 RTM_ADD: Add Route: len 192, priority 146, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 428 RTM_ADD: Add Route: len 192, priority 18, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 43 RTM_DELETE: Delete Route: len 192, priority 146, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 478 RTM_DELETE: Delete Route: len 192, priority 18, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 31 RTM_IFINFO: iface status change: len 168, if# 6, name carp1, link: backup, mtu: 1500, flags: 31 RTM_IFINFO: iface status change: len 168, if# 6, name carp1, link: invalid, mtu: 1500, flags: 31 RTM_IFINFO: iface status change: len 168, if# 6, name carp1, link: master, mtu: 1500, flags: 1 RTM_RESOLVE: Route created by cloning: len 192, priority 146, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 385 RTM_RESOLVE: Route created by cloning: len 192, priority 18, table 0, if# 6, name carp1, pid: 0, seq 0, errno 0 erl3-01$ grep vlan100 route.monitor | sort | uniq -c 31 RTM_IFINFO: iface status change: len 168, if# 8, name vlan100, link: active, mtu: 1500, flags: 31 RTM_IFINFO: iface status change: len 168, if# 8, name vlan100, link: no carrier, mtu: 1500, flags: erl3-01$ grep cnmac2 route.monitor | sort | uniq -c 57 RTM_ADD: Add Route: len 192, priority 3, table 0, if# 3, name cnmac2, pid: 0, seq 0, errno 0 57 RTM_DELETE: Delete Route: len 192, priority 3, table 0, if# 3, name cnmac2, pid: 0, seq 0, errno 0 31 RTM_IFINFO: iface status change: len 168, if# 3, name cnmac2, link: active, mtu: 1500, flags: 31 RTM_IFINFO: iface status change: len 168, if# 3, name cnmac2, link: no carrier, mtu: 1500, flags: It looks like the underlying cnmac2 interface is flapping...so, that's a bummer. As generally underpowered as this machine is, might the kernel be overwhelmed with other tasks, and have a watchdog timeout mark the cnmac2 interface as down (due to some expired timeout)? Just grasping for something here...my next steps are to swap this unit out with the other one (to try and eliminate hardware failure of THIS unit). Any other suggestions? On Mon, Feb 1, 2021 at 3:04 AM David Gwynne wrote: > > > > > On 1 Feb 2021, at 6:02 pm, Bryan Stenson wrote: > > > > Hi all - > > > > I'm trying to setup a pair of ERL3 octeon routers in master/standby > > mode via carp/pfsync to route traffic from my internal lan to the > > internet. I've seen strange behavior wrt carp on these machines, so > > in an attempt to reduce the problem, I've removed one completely. > > > > Even with only a single box (ERL3-01) on the network configured as a > > carp member, the carp interface state periodically changes (as seen > > from ifstated(8)). > > > > I'm wondering if disconnecting the other ERL3 device is a valid isolated > > test. > > 1. Will/might this cause issues with the carp device, as it cannot > > determine state from any other host? > > If carp state flaps around while it is the only device on the network, that > would imply the parent device is flapping around. > > > 2. Will/might this cause issues as it cannot send/receive pfsync > > updates (the other node is disconnected). > > pfsync doesn't really care about carp state. > > > 3. Is there something else in my setup causing carp to fail here? > > I'd be running "route monitor" and looking for link state changes on the carp > parent interface. > > > 4. Could this be hardware/temperature related to this ERL3? Wouldn't > > I see an additional error in dmesg if the physical device (cnmac2) > > failed periodically? > > > > I'd appreciate any pointers here...I feel like I'm missing something dumb. > > My first ideas are above. If it turns out the carp parent is stable we can > try come up with something else. > > dlg > > > > > Thanks in advance. > > > > Bryan > > > > Here are some of my configs. If I've missed including something > > critical to help describe my setup, please let me know and I'll add > > it. > > > > ## Help me OBSD-Misc Kenobi. You're my only hope. ## > > > > erl3-01# uname -a > > OpenBSD erl3-01.siliconvortex.com 6.8 GENERIC#522 octeon > > > > erl3-01# dmesg > > ... > > carp1: state transition: BACKUP -> MASTER > > carp1: state transition: BACKUP -> MASTER > > carp1: state transition: BACKUP -> MASTER > > carp1: state transition: BACKUP -> MASTER > > carp1: state transition: BACKUP -> MASTER > > carp1: state transition
Re: seeing carp interface state change for unknown reason ; cluestick hunting
> On 1 Feb 2021, at 6:02 pm, Bryan Stenson wrote:
>
> Hi all -
>
> I'm trying to setup a pair of ERL3 octeon routers in master/standby
> mode via carp/pfsync to route traffic from my internal lan to the
> internet. I've seen strange behavior wrt carp on these machines, so
> in an attempt to reduce the problem, I've removed one completely.
>
> Even with only a single box (ERL3-01) on the network configured as a
> carp member, the carp interface state periodically changes (as seen
> from ifstated(8)).
>
> I'm wondering if disconnecting the other ERL3 device is a valid isolated test.
> 1. Will/might this cause issues with the carp device, as it cannot
> determine state from any other host?
If carp state flaps around while it is the only device on the network, that
would imply the parent device is flapping around.
> 2. Will/might this cause issues as it cannot send/receive pfsync
> updates (the other node is disconnected).
pfsync doesn't really care about carp state.
> 3. Is there something else in my setup causing carp to fail here?
I'd be running "route monitor" and looking for link state changes on the carp
parent interface.
> 4. Could this be hardware/temperature related to this ERL3? Wouldn't
> I see an additional error in dmesg if the physical device (cnmac2)
> failed periodically?
>
> I'd appreciate any pointers here...I feel like I'm missing something dumb.
My first ideas are above. If it turns out the carp parent is stable we can try
come up with something else.
dlg
>
> Thanks in advance.
>
> Bryan
>
> Here are some of my configs. If I've missed including something
> critical to help describe my setup, please let me know and I'll add
> it.
>
> ## Help me OBSD-Misc Kenobi. You're my only hope. ##
>
> erl3-01# uname -a
> OpenBSD erl3-01.siliconvortex.com 6.8 GENERIC#522 octeon
>
> erl3-01# dmesg
> ...
> carp1: state transition: BACKUP -> MASTER
> carp1: state transition: BACKUP -> MASTER
> carp1: state transition: BACKUP -> MASTER
> carp1: state transition: BACKUP -> MASTER
> carp1: state transition: BACKUP -> MASTER
> carp1: state transition: BACKUP -> MASTER
>
> erl3-01# tail mbox
> Mon, 1 Feb 2021 06:49:26 + (UTC)
> From: Charlie Root
> Date: Mon, 1 Feb 2021 06:49:25 + (UTC)
> To: root@localhost
> Subject: carp master changed
> Message-ID: <515eb74cff427...@erl3-01.siliconvortex.com>
> Status: RO
>
> master is now erl3-01.siliconvortex.com
>
>
> erl3-01# sysctl -a | grep carp
> net.inet.carp.allow=1
> net.inet.carp.preempt=1
> net.inet.carp.log=2
>
> erl3-01# cat /etc/hostname.carp1
> #carp for lan side
> 192.168.122.1/23 carpdev vlan100 vhid 1 pass somethinglongandsecret
>
> erl3-01# cat /etc/hostname.vlan100
> vnetid 100 parent cnmac2
> up
>
> erl3-01# cat /etc/hostname.cnmac2
> inet 192.168.1.253 255.255.254.0
>
> erl3-01# cat /etc/hostname.pfsync0
> up syncdev cnmac1
>
> erl3-01# cat /etc/hostname.cnmac1
> inet 10.10.200.1 255.255.255.252
>
> erl3-01# cat /etc/ifstated.conf
> # Initial State
> init-state auto
>
> # Macros
> if_carp_up="carp1.link.up"
> if_carp_down="!carp1.link.up"
>
> state auto {
> if $if_carp_up {
>set-state master
> }
>
> if $if_carp_down {
>set-state backup
> }
> }
>
> state master {
> init {
>run "echo master is now `hostname` | mail -s 'carp master changed'
> root@localhost"
> }
>
> if $if_carp_down {
>set-state backup
> }
> }
>
> state backup {
> init {
>run "echo backup is now `hostname` | mail -s 'carp master changed
> root@localhost"
> }
>
> if $if_carp_up {
>set-state master
> }
> }
>
> erl3-01# cat /etc/pf.conf
> # adopted from https://www.openbsd.org/faq/pf/example1.html
> wan_dev = cnmac0
> lan_dev = cnmac2
> carp_dev = vlan100
> pfsync_dev = cnmac1
> table { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
>172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
>192.168.0.0/16 198.18.0.0/15 198.51.100.0/24\
>203.0.113.0/24 }
>
> # carp
> pass quick on $lan_dev proto carp keep state (no-sync)
>
> # pfsync
> pass quick on $pfsync_dev proto pfsync keep state (no-sync)
>
> set block-policy drop
> set loginterface $wan_dev
> set skip on lo0
>
> match in all scrub (no-df random-id max-mss 1440)
>
> # redirect DNS queries to localhost
> pass in quick on { $carp_dev $lan_dev } proto { udp tcp } from any to
> any port domain rdr-to 192.168.1.253 port domain
>
> # NAT to the world
> match out on $wan_dev inet from !($wan_dev:network) to any nat-to ($wan_dev:0)
>
> antispoof quick for { $wan_dev }
>
> # martians
> block in quick on $wan_dev from to any
> block return out quick on $wan_dev from any to
>
> block all
>
> # manage buffer bloat
> queue outq on $wan_dev flows 1024 bandwidth 3M max 3M qlimit 1024 default
> queue inq on $lan_dev flows 1024 bandwidth 45M max 45M qlimit 1024 default
>
> pass out quick inet
>
> pass in on { $carp_dev $lan_dev } inet
>
seeing carp interface state change for unknown reason ; cluestick hunting
Hi all -
I'm trying to setup a pair of ERL3 octeon routers in master/standby
mode via carp/pfsync to route traffic from my internal lan to the
internet. I've seen strange behavior wrt carp on these machines, so
in an attempt to reduce the problem, I've removed one completely.
Even with only a single box (ERL3-01) on the network configured as a
carp member, the carp interface state periodically changes (as seen
from ifstated(8)).
I'm wondering if disconnecting the other ERL3 device is a valid isolated test.
1. Will/might this cause issues with the carp device, as it cannot
determine state from any other host?
2. Will/might this cause issues as it cannot send/receive pfsync
updates (the other node is disconnected).
3. Is there something else in my setup causing carp to fail here?
4. Could this be hardware/temperature related to this ERL3? Wouldn't
I see an additional error in dmesg if the physical device (cnmac2)
failed periodically?
I'd appreciate any pointers here...I feel like I'm missing something dumb.
Thanks in advance.
Bryan
Here are some of my configs. If I've missed including something
critical to help describe my setup, please let me know and I'll add
it.
## Help me OBSD-Misc Kenobi. You're my only hope. ##
erl3-01# uname -a
OpenBSD erl3-01.siliconvortex.com 6.8 GENERIC#522 octeon
erl3-01# dmesg
...
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
erl3-01# tail mbox
Mon, 1 Feb 2021 06:49:26 + (UTC)
From: Charlie Root
Date: Mon, 1 Feb 2021 06:49:25 + (UTC)
To: root@localhost
Subject: carp master changed
Message-ID: <515eb74cff427...@erl3-01.siliconvortex.com>
Status: RO
master is now erl3-01.siliconvortex.com
erl3-01# sysctl -a | grep carp
net.inet.carp.allow=1
net.inet.carp.preempt=1
net.inet.carp.log=2
erl3-01# cat /etc/hostname.carp1
#carp for lan side
192.168.122.1/23 carpdev vlan100 vhid 1 pass somethinglongandsecret
erl3-01# cat /etc/hostname.vlan100
vnetid 100 parent cnmac2
up
erl3-01# cat /etc/hostname.cnmac2
inet 192.168.1.253 255.255.254.0
erl3-01# cat /etc/hostname.pfsync0
up syncdev cnmac1
erl3-01# cat /etc/hostname.cnmac1
inet 10.10.200.1 255.255.255.252
erl3-01# cat /etc/ifstated.conf
# Initial State
init-state auto
# Macros
if_carp_up="carp1.link.up"
if_carp_down="!carp1.link.up"
state auto {
if $if_carp_up {
set-state master
}
if $if_carp_down {
set-state backup
}
}
state master {
init {
run "echo master is now `hostname` | mail -s 'carp master changed'
root@localhost"
}
if $if_carp_down {
set-state backup
}
}
state backup {
init {
run "echo backup is now `hostname` | mail -s 'carp master changed
root@localhost"
}
if $if_carp_up {
set-state master
}
}
erl3-01# cat /etc/pf.conf
# adopted from https://www.openbsd.org/faq/pf/example1.html
wan_dev = cnmac0
lan_dev = cnmac2
carp_dev = vlan100
pfsync_dev = cnmac1
table { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24\
203.0.113.0/24 }
# carp
pass quick on $lan_dev proto carp keep state (no-sync)
# pfsync
pass quick on $pfsync_dev proto pfsync keep state (no-sync)
set block-policy drop
set loginterface $wan_dev
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
# redirect DNS queries to localhost
pass in quick on { $carp_dev $lan_dev } proto { udp tcp } from any to
any port domain rdr-to 192.168.1.253 port domain
# NAT to the world
match out on $wan_dev inet from !($wan_dev:network) to any nat-to ($wan_dev:0)
antispoof quick for { $wan_dev }
# martians
block in quick on $wan_dev from to any
block return out quick on $wan_dev from any to
block all
# manage buffer bloat
queue outq on $wan_dev flows 1024 bandwidth 3M max 3M qlimit 1024 default
queue inq on $lan_dev flows 1024 bandwidth 45M max 45M qlimit 1024 default
pass out quick inet
pass in on { $carp_dev $lan_dev } inet
Re: CARP load balancing problems under KVM
Many thanks for your help Giannis ... I am not using oVirt to manage this KVM host, only default installed tools: libvirtd, virsh ... In any case there is not any filter applied in libvirtd On 12/1/21, 20:13, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" wrote: On 12/01/2021 18:58, Carlos Lopez wrote: > Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC spoofing is avoided using this option: > > bridge.ageing-time: 300 > > On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" wrote: > > Check that you have mac spoofing filter disabled on that interface. For carp to work, I have it disabled on the virtual interface on the supervisor. I use ovirt so I do it on the network profile there. This is a libvirt filter option. Maybe it's applied by default on your setup. The idea is to disable this protection and allow mac spoofing. G
Re: CARP load balancing problems under KVM
On 12/01/2021 18:58, Carlos Lopez wrote: Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC spoofing is avoided using this option: bridge.ageing-time: 300 On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis" wrote: Check that you have mac spoofing filter disabled on that interface. For carp to work, I have it disabled on the virtual interface on the supervisor. I use ovirt so I do it on the network profile there. This is a libvirt filter option. Maybe it's applied by default on your setup. The idea is to disable this protection and allow mac spoofing. G
Re: CARP load balancing problems under KVM
Thanks Gianni, but about what interface ? KVM bridges? In theory, MAC spoofing
is avoided using this option:
bridge.ageing-time: 300
On 12/1/21, 17:47, "owner-m...@openbsd.org on behalf of Kapetanakis Giannis"
wrote:
Check that you have mac spoofing filter disabled on that interface.
G
On 12/01/2021 15:30, Carlos Lopez wrote:
> Hi David and misc@,
>
> Sorry to disturb with this.I have realized several tests this morning
with two OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result
is the same: carp load balancing doesn't work. My host is a RedHat Enterprise
Linux 8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have
tested all ip load balancing options under these OpenBSD virtual guests: ip,
ip-unicast and ip-stealth.
>
> When I use only "ip" for load balancing in all carped interfaces, these
appears as MASTER in both firewalls as you can see here:
>
> carp0: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:0a
> description: Production Network
> index 12 priority 15 llprio 3
> carp: carpdev vio0 advbase 1 balancing ip
> state MASTER vhid 10 advskew 0
> state MASTER vhid 11 advskew 100
> groups: carp
> status: master
> inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
> carp1: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:0c
> description: Primary Public Network
> index 13 priority 15 llprio 3
> carp: carpdev vio1 advbase 1 balancing ip
> state MASTER vhid 12 advskew 0
> state MASTER vhid 13 advskew 100
> groups: carp
> status: master
> inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
> carp2: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:0e
> description: Mgmt Network
> index 14 priority 15 llprio 3
> carp: carpdev vio2 advbase 1 balancing ip
> state MASTER vhid 14 advskew 0
> state MASTER vhid 15 advskew 100
> groups: carp
> status: master
> inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
> carp3: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:10
> description: DMZ Network
> index 15 priority 15 llprio 3
> carp: carpdev vio3 advbase 1 balancing ip
> state MASTER vhid 16 advskew 0
> state MASTER vhid 17 advskew 100
> groups: carp
> status: master
> inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
> carp4: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:12
> description: VPN Network
> index 16 priority 15 llprio 3
> carp: carpdev vio4 advbase 1 balancing ip
> state MASTER vhid 18 advskew 0
> state MASTER vhid 19 advskew 100
> groups: carp
> status: master
> inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
> carp5: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:14
> description: Encryption Network
> index 17 priority 15 llprio 3
> carp: carpdev vio5 advbase 1 balancing ip
> state MASTER vhid 20 advskew 0
> state MASTER vhid 21 advskew 100
> groups: carp
> status: master
> inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
> carp6: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:16
> description: IDPS Network
> index 18 priority 15 llprio 3
> carp: carpdev vio6 advbase 1 balancing ip
> state MASTER vhid 22 advskew 0
> state MASTER vhid 23 advskew 100
> groups: carp
> status: master
> inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
> carp7: flags=8843 mtu 1500
> lladdr 00:00:5e:00:01:18
> description: Windows Network
> index 19 priority 15 llprio 3
> carp: carpdev vio8 advbase 1 balancing ip
> state MASTER vhid 24 advskew 0
> state MASTER vhid 25 advskew 100
> groups: carp
> status: master
> inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7
>
> .. and all kvm guests behind these firewalls shows the following arp
entries (172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01
and 172.22.55.29 is for fw02):
>
> root@stonehenge:~# ip neigh
> 172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
> 172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
> 172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
> 172.22.55.30 dev eth0 INCOMPLETE
> 172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
> 172.22.55.4 dev eth0 ll
Re: CARP load balancing problems under KVM
Check that you have mac spoofing filter disabled on that interface.
G
On 12/01/2021 15:30, Carlos Lopez wrote:
Hi David and misc@,
Sorry to disturb with this.I have realized several tests this morning with two
OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result is the
same: carp load balancing doesn't work. My host is a RedHat Enterprise Linux
8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have tested
all ip load balancing options under these OpenBSD virtual guests: ip,
ip-unicast and ip-stealth.
When I use only "ip" for load balancing in all carped interfaces, these appears
as MASTER in both firewalls as you can see here:
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip
state MASTER vhid 10 advskew 0
state MASTER vhid 11 advskew 100
groups: carp
status: master
inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
carp1: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0c
description: Primary Public Network
index 13 priority 15 llprio 3
carp: carpdev vio1 advbase 1 balancing ip
state MASTER vhid 12 advskew 0
state MASTER vhid 13 advskew 100
groups: carp
status: master
inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
carp2: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0e
description: Mgmt Network
index 14 priority 15 llprio 3
carp: carpdev vio2 advbase 1 balancing ip
state MASTER vhid 14 advskew 0
state MASTER vhid 15 advskew 100
groups: carp
status: master
inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
carp3: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:10
description: DMZ Network
index 15 priority 15 llprio 3
carp: carpdev vio3 advbase 1 balancing ip
state MASTER vhid 16 advskew 0
state MASTER vhid 17 advskew 100
groups: carp
status: master
inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
carp4: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:12
description: VPN Network
index 16 priority 15 llprio 3
carp: carpdev vio4 advbase 1 balancing ip
state MASTER vhid 18 advskew 0
state MASTER vhid 19 advskew 100
groups: carp
status: master
inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
carp5: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:14
description: Encryption Network
index 17 priority 15 llprio 3
carp: carpdev vio5 advbase 1 balancing ip
state MASTER vhid 20 advskew 0
state MASTER vhid 21 advskew 100
groups: carp
status: master
inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
carp6: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:16
description: IDPS Network
index 18 priority 15 llprio 3
carp: carpdev vio6 advbase 1 balancing ip
state MASTER vhid 22 advskew 0
state MASTER vhid 23 advskew 100
groups: carp
status: master
inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
carp7: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:18
description: Windows Network
index 19 priority 15 llprio 3
carp: carpdev vio8 advbase 1 balancing ip
state MASTER vhid 24 advskew 0
state MASTER vhid 25 advskew 100
groups: carp
status: master
inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7
.. and all kvm guests behind these firewalls shows the following arp entries
(172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01 and
172.22.55.29 is for fw02):
root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0 INCOMPLETE
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 REACHABLE
And after some seconds, arp entries appears as:
root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0 FAILED
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa REACHABLE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 DELAY
In this case, no traffic flows and I can't see any packet using tcpdump
When I use "ip-unicast" or "ip-stealth" options for CARP load balancing, CARP
interfaces seems to be ok:
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
d
Re: CARP load balancing problems under KVM
Hi David and misc@,
Sorry to disturb with this.I have realized several tests this morning with two
OpenBSD 6.8 carp'ed firewalls (fully patched) as kvm guests and result is the
same: carp load balancing doesn't work. My host is a RedHat Enterprise Linux
8.3 with kernel .18.0-240.10.1.el8_3.x86_6 (fully patched also). I have tested
all ip load balancing options under these OpenBSD virtual guests: ip,
ip-unicast and ip-stealth.
When I use only "ip" for load balancing in all carped interfaces, these appears
as MASTER in both firewalls as you can see here:
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip
state MASTER vhid 10 advskew 0
state MASTER vhid 11 advskew 100
groups: carp
status: master
inet 172.22.55.30 netmask 0xffe0 broadcast 172.22.55.31
carp1: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0c
description: Primary Public Network
index 13 priority 15 llprio 3
carp: carpdev vio1 advbase 1 balancing ip
state MASTER vhid 12 advskew 0
state MASTER vhid 13 advskew 100
groups: carp
status: master
inet 172.17.35.5 netmask 0xff00 broadcast 172.17.35.255
carp2: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0e
description: Mgmt Network
index 14 priority 15 llprio 3
carp: carpdev vio2 advbase 1 balancing ip
state MASTER vhid 14 advskew 0
state MASTER vhid 15 advskew 100
groups: carp
status: master
inet 172.22.59.1 netmask 0xfff0 broadcast 172.22.59.15
carp3: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:10
description: DMZ Network
index 15 priority 15 llprio 3
carp: carpdev vio3 advbase 1 balancing ip
state MASTER vhid 16 advskew 0
state MASTER vhid 17 advskew 100
groups: carp
status: master
inet 172.22.54.1 netmask 0xfff8 broadcast 172.22.54.7
carp4: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:12
description: VPN Network
index 16 priority 15 llprio 3
carp: carpdev vio4 advbase 1 balancing ip
state MASTER vhid 18 advskew 0
state MASTER vhid 19 advskew 100
groups: carp
status: master
inet 172.22.56.1 netmask 0xfff8 broadcast 172.22.56.7
carp5: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:14
description: Encryption Network
index 17 priority 15 llprio 3
carp: carpdev vio5 advbase 1 balancing ip
state MASTER vhid 20 advskew 0
state MASTER vhid 21 advskew 100
groups: carp
status: master
inet 172.22.57.1 netmask 0xfff8 broadcast 172.22.57.7
carp6: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:16
description: IDPS Network
index 18 priority 15 llprio 3
carp: carpdev vio6 advbase 1 balancing ip
state MASTER vhid 22 advskew 0
state MASTER vhid 23 advskew 100
groups: carp
status: master
inet 172.22.60.1 netmask 0xfff0 broadcast 172.22.60.15
carp7: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:18
description: Windows Network
index 19 priority 15 llprio 3
carp: carpdev vio8 advbase 1 balancing ip
state MASTER vhid 24 advskew 0
state MASTER vhid 25 advskew 100
groups: carp
status: master
inet 172.22.61.1 netmask 0xfff8 broadcast 172.22.61.7
.. and all kvm guests behind these firewalls shows the following arp entries
(172.22.55.30 is the carp'ed IP address. IP 172.22.55.28 is for fw01 and
172.22.55.29 is for fw02):
root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0 INCOMPLETE
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa STALE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 REACHABLE
And after some seconds, arp entries appears as:
root@stonehenge:~# ip neigh
172.22.55.14 dev eth0 lladdr ac:1f:6b:14:6a:f6 STALE
172.22.55.29 dev eth0 lladdr 00:50:56:a1:4d:c3 STALE
172.22.55.5 dev eth0 lladdr ac:1f:6b:14:6c:b0 STALE
172.22.55.30 dev eth0 FAILED
172.22.55.28 dev eth0 lladdr 00:50:56:6f:64:aa REACHABLE
172.22.55.4 dev eth0 lladdr a0:ce:c8:0a:7e:f4 DELAY
In this case, no traffic flows and I can't see any packet using tcpdump
When I use "ip-unicast" or "ip-stealth" options for CARP load balancing, CARP
interfaces seems to be ok:
carp0: flags=8843 mtu 1500
lladdr 00:00:5e:00:01:0a
description: Production Network
index 12 priority 15 llprio 3
carp: carpdev vio0 advbase 1 balancing ip-stealt
Re: OSPF and CARP interfaces
Hello,
The fix recommended by Remi works great. Can we have this into an official
patch?
Why I am not using it as Claudio recommends is that vlan20 in my case also
is a transit vlan like vlan21 so it cannot be a passive interface. From the
docs I understand that having carp listed as an interface will force it
into passive mode. I should have mentioned this in my original email, sorry
about that.
All in all, current snapshot seems to do what documentation says in terms
of "depend on". The stable 6.8 does not so a patch is warranted I think.
On Tue, Dec 22, 2020 at 3:50 PM Claudio Jeker
wrote:
> On Tue, Dec 22, 2020 at 02:04:27PM +0100, open...@kene.nu wrote:
> > Hello,
> > I am seeing what I deem to be unexpected behavior with ospfd and
> depending
> > on carp interfaces.
> > Running 6.8 with latest patches applied on all three routers.
> >
> > # uname -a
> > OpenBSD extfw1.lab.kambi.com 6.8 GENERIC.MP#2 amd64
> >
> > My setup is as following;
> > Two openbsd boxes (FW1 and FW2) acting as a firewall pair sharing carp
> > interfaces.
> > Single openbsd box (R1) that in this instance acts as a client trying to
> > reach servers that are reachable via the FWs.
> > VLan20 (actually carp20) is my nexthop (BGP wise) to reach any networks
> > behind the FW pair.
> > VLan21 is the link network between all the three boxes. The FWs share a
> > carp21 interface.
> >
> > My FW ospfd.conf (same on all three boxes apart from the "depend on"
> which
> > is absent from R1):
> > router-id
> >
> > area 0.0.0.0 {
> > interface lo1
> > interface vlan20 {
> > depend on carp20
> > }
> > interface vlan21 {
> > depend on carp21
> > }
> > }
>
> I would change the config to just use
>
> area 0.0.0.0 {
> interface lo1
> interface carp20
> interface vlan21
> }
>
> This way the network on vlan20/carp20 will be announced depending on the
> carp state with the backup system announcing the same route with a high
> metric. There is no need to use "depend on" for such a simple case.
>
> For vlan21 I would not do that since there you want reachability in any
> case especially if you announce BGP networks on the firewalls with the
> carp21 address (instead of the default vlan21 one).
>
> > Carp20:
> > root@FW1:~ # ifconfig carp20 | grep inet
> > inet 172.30.9.21 netmask 0xfff0 broadcast 172.30.9.31
> >
> > Now to the strange part. I see that the selected route in R1 points to
> FW1
> > even though carp20/21 on FW1 is in state BACKUP. No matter what I do,
> apart
> > from setting static metrics, ospfd on R1 always selects FW1 as nexthop.
> > root@FW1:~ # ifconfig vlan21 | grep inet
> > inet 172.30.9.34 netmask 0xfff0 broadcast 172.30.9.47
> > root@FW1:~ # ifconfig carp20 | grep carp:
> > carp: BACKUP carpdev vlan20 vhid 1 advbase 1 advskew 10
> > root@FW1:~ # ifconfig carp21 | grep carp:
> > carp: BACKUP carpdev vlan21 vhid 1 advbase 1 advskew 10
> >
> > root@FW2:~ # ifconfig vlan21 | grep inet
> > inet 172.30.9.35 netmask 0xfff0 broadcast 172.30.9.47
> > root@FW2:~ # ifconfig carp20 | grep carp:
> > carp: MASTER carpdev vlan20 vhid 1 advbase 1 advskew 100
> > root@FW2:~ # ifconfig carp21 | grep carp:
> > carp: MASTER carpdev vlan21 vhid 1 advbase 1 advskew 100
> >
> > root@R1:~ # ospfctl sh
> > neighID Pri StateDeadTime Address Iface
> > Uptime
> > 172.30.9.4 1 FULL/OTHER 00:00:38 172.30.9.35 vlan21
> 00:21:33
> > 172.30.9.3 1 FULL/BCKUP 00:00:38 172.30.9.34 vlan21
> 00:22:14
> >
> > root@R1:~ # ospfctl sh fib | grep 172.30.9.16/2
> > *O 32 172.30.9.16/28 172.30.9.34
> > *O 32 172.30.9.16/28 172.30.9.35
> >
> > root@R1:~ # ospfctl sh rib | grep 172.30.9.16/2
> > 172.30.9.16/28 172.30.9.34 Intra-Area Network 20
> > 00:30:33
> > 172.30.9.16/28 172.30.9.35 Intra-Area Network 20
> > 00:29:56
> >
> > root@R1:~ # route -n get 172.30.9.21
> >route to: 172.30.9.21
> > destination: 172.30.9.16
> >mask: 255.255.255.240
> > gateway: 172.30.9.34
> > interface: vlan21
> > if address: 172.30.9.37
> >priority: 32 (ospf)
> > flags:
> > use mtuexpire
> > 11 0 0
> >
> > As seen above R1 selects 172.30.9.34 as the nexthop based on ospf which
> is
> > wrong. It should be 172.30.9.35 as FW2 is carp master for carp20/21.
> What I
> > in the end want to achieve is that the router with carp20/21 MASTER
> should
> > be the preferred carp20 nexthop. An assumption can be made that carp20/21
> > will always have the same FW as master in my case.
>
> --
> :wq Claudio
>
Re: OSPF and CARP interfaces
On Tue, Dec 22, 2020 at 02:04:27PM +0100, open...@kene.nu wrote:
> Hello,
> I am seeing what I deem to be unexpected behavior with ospfd and depending
> on carp interfaces.
> Running 6.8 with latest patches applied on all three routers.
>
> # uname -a
> OpenBSD extfw1.lab.kambi.com 6.8 GENERIC.MP#2 amd64
>
> My setup is as following;
> Two openbsd boxes (FW1 and FW2) acting as a firewall pair sharing carp
> interfaces.
> Single openbsd box (R1) that in this instance acts as a client trying to
> reach servers that are reachable via the FWs.
> VLan20 (actually carp20) is my nexthop (BGP wise) to reach any networks
> behind the FW pair.
> VLan21 is the link network between all the three boxes. The FWs share a
> carp21 interface.
>
> My FW ospfd.conf (same on all three boxes apart from the "depend on" which
> is absent from R1):
> router-id
>
> area 0.0.0.0 {
> interface lo1
> interface vlan20 {
> depend on carp20
> }
> interface vlan21 {
> depend on carp21
> }
> }
I would change the config to just use
area 0.0.0.0 {
interface lo1
interface carp20
interface vlan21
}
This way the network on vlan20/carp20 will be announced depending on the
carp state with the backup system announcing the same route with a high
metric. There is no need to use "depend on" for such a simple case.
For vlan21 I would not do that since there you want reachability in any
case especially if you announce BGP networks on the firewalls with the
carp21 address (instead of the default vlan21 one).
> Carp20:
> root@FW1:~ # ifconfig carp20 | grep inet
> inet 172.30.9.21 netmask 0xfff0 broadcast 172.30.9.31
>
> Now to the strange part. I see that the selected route in R1 points to FW1
> even though carp20/21 on FW1 is in state BACKUP. No matter what I do, apart
> from setting static metrics, ospfd on R1 always selects FW1 as nexthop.
> root@FW1:~ # ifconfig vlan21 | grep inet
> inet 172.30.9.34 netmask 0xfff0 broadcast 172.30.9.47
> root@FW1:~ # ifconfig carp20 | grep carp:
> carp: BACKUP carpdev vlan20 vhid 1 advbase 1 advskew 10
> root@FW1:~ # ifconfig carp21 | grep carp:
> carp: BACKUP carpdev vlan21 vhid 1 advbase 1 advskew 10
>
> root@FW2:~ # ifconfig vlan21 | grep inet
> inet 172.30.9.35 netmask 0xfff0 broadcast 172.30.9.47
> root@FW2:~ # ifconfig carp20 | grep carp:
> carp: MASTER carpdev vlan20 vhid 1 advbase 1 advskew 100
> root@FW2:~ # ifconfig carp21 | grep carp:
> carp: MASTER carpdev vlan21 vhid 1 advbase 1 advskew 100
>
> root@R1:~ # ospfctl sh
> neighID Pri StateDeadTime Address Iface
> Uptime
> 172.30.9.4 1 FULL/OTHER 00:00:38 172.30.9.35 vlan2100:21:33
> 172.30.9.3 1 FULL/BCKUP 00:00:38 172.30.9.34 vlan2100:22:14
>
> root@R1:~ # ospfctl sh fib | grep 172.30.9.16/2
> *O 32 172.30.9.16/28 172.30.9.34
> *O 32 172.30.9.16/28 172.30.9.35
>
> root@R1:~ # ospfctl sh rib | grep 172.30.9.16/2
> 172.30.9.16/28 172.30.9.34 Intra-Area Network 20
> 00:30:33
> 172.30.9.16/28 172.30.9.35 Intra-Area Network 20
> 00:29:56
>
> root@R1:~ # route -n get 172.30.9.21
>route to: 172.30.9.21
> destination: 172.30.9.16
>mask: 255.255.255.240
> gateway: 172.30.9.34
> interface: vlan21
> if address: 172.30.9.37
>priority: 32 (ospf)
> flags:
> use mtuexpire
> 11 0 0
>
> As seen above R1 selects 172.30.9.34 as the nexthop based on ospf which is
> wrong. It should be 172.30.9.35 as FW2 is carp master for carp20/21. What I
> in the end want to achieve is that the router with carp20/21 MASTER should
> be the preferred carp20 nexthop. An assumption can be made that carp20/21
> will always have the same FW as master in my case.
--
:wq Claudio
Re: OSPF and CARP interfaces
On Tue, Dec 22, 2020 at 02:04:27PM +0100, open...@kene.nu wrote:
> Hello,
> I am seeing what I deem to be unexpected behavior with ospfd and depending
> on carp interfaces.
> Running 6.8 with latest patches applied on all three routers.
>
> # uname -a
> OpenBSD extfw1.lab.kambi.com 6.8 GENERIC.MP#2 amd64
>
> My setup is as following;
> Two openbsd boxes (FW1 and FW2) acting as a firewall pair sharing carp
> interfaces.
> Single openbsd box (R1) that in this instance acts as a client trying to
> reach servers that are reachable via the FWs.
> VLan20 (actually carp20) is my nexthop (BGP wise) to reach any networks
> behind the FW pair.
> VLan21 is the link network between all the three boxes. The FWs share a
> carp21 interface.
>
> My FW ospfd.conf (same on all three boxes apart from the "depend on" which
> is absent from R1):
> router-id
>
> area 0.0.0.0 {
> interface lo1
> interface vlan20 {
> depend on carp20
> }
> interface vlan21 {
> depend on carp21
> }
> }
>
> Carp20:
> root@FW1:~ # ifconfig carp20 | grep inet
> inet 172.30.9.21 netmask 0xfff0 broadcast 172.30.9.31
>
> Now to the strange part. I see that the selected route in R1 points to FW1
> even though carp20/21 on FW1 is in state BACKUP. No matter what I do, apart
> from setting static metrics, ospfd on R1 always selects FW1 as nexthop.
> root@FW1:~ # ifconfig vlan21 | grep inet
> inet 172.30.9.34 netmask 0xfff0 broadcast 172.30.9.47
> root@FW1:~ # ifconfig carp20 | grep carp:
> carp: BACKUP carpdev vlan20 vhid 1 advbase 1 advskew 10
> root@FW1:~ # ifconfig carp21 | grep carp:
> carp: BACKUP carpdev vlan21 vhid 1 advbase 1 advskew 10
>
> root@FW2:~ # ifconfig vlan21 | grep inet
> inet 172.30.9.35 netmask 0xfff0 broadcast 172.30.9.47
> root@FW2:~ # ifconfig carp20 | grep carp:
> carp: MASTER carpdev vlan20 vhid 1 advbase 1 advskew 100
> root@FW2:~ # ifconfig carp21 | grep carp:
> carp: MASTER carpdev vlan21 vhid 1 advbase 1 advskew 100
>
> root@R1:~ # ospfctl sh
> neighID Pri StateDeadTime Address Iface
> Uptime
> 172.30.9.4 1 FULL/OTHER 00:00:38 172.30.9.35 vlan2100:21:33
> 172.30.9.3 1 FULL/BCKUP 00:00:38 172.30.9.34 vlan2100:22:14
>
> root@R1:~ # ospfctl sh fib | grep 172.30.9.16/2
> *O 32 172.30.9.16/28 172.30.9.34
> *O 32 172.30.9.16/28 172.30.9.35
>
> root@R1:~ # ospfctl sh rib | grep 172.30.9.16/2
> 172.30.9.16/28 172.30.9.34 Intra-Area Network 20
> 00:30:33
> 172.30.9.16/28 172.30.9.35 Intra-Area Network 20
> 00:29:56
>
> root@R1:~ # route -n get 172.30.9.21
>route to: 172.30.9.21
> destination: 172.30.9.16
>mask: 255.255.255.240
> gateway: 172.30.9.34
> interface: vlan21
> if address: 172.30.9.37
>priority: 32 (ospf)
> flags:
> use mtuexpire
> 11 0 0
>
> As seen above R1 selects 172.30.9.34 as the nexthop based on ospf which is
> wrong. It should be 172.30.9.35 as FW2 is carp master for carp20/21. What I
> in the end want to achieve is that the router with carp20/21 MASTER should
> be the preferred carp20 nexthop. An assumption can be made that carp20/21
> will always have the same FW as master in my case.
Can you test if it works as expected with current?
I think you are affected by a bug fixed by dlg with this commit:
https://marc.info/?l=openbsd-cvs=160427701605657=2
OSPF and CARP interfaces
Hello,
I am seeing what I deem to be unexpected behavior with ospfd and depending
on carp interfaces.
Running 6.8 with latest patches applied on all three routers.
# uname -a
OpenBSD extfw1.lab.kambi.com 6.8 GENERIC.MP#2 amd64
My setup is as following;
Two openbsd boxes (FW1 and FW2) acting as a firewall pair sharing carp
interfaces.
Single openbsd box (R1) that in this instance acts as a client trying to
reach servers that are reachable via the FWs.
VLan20 (actually carp20) is my nexthop (BGP wise) to reach any networks
behind the FW pair.
VLan21 is the link network between all the three boxes. The FWs share a
carp21 interface.
My FW ospfd.conf (same on all three boxes apart from the "depend on" which
is absent from R1):
router-id
area 0.0.0.0 {
interface lo1
interface vlan20 {
depend on carp20
}
interface vlan21 {
depend on carp21
}
}
Carp20:
root@FW1:~ # ifconfig carp20 | grep inet
inet 172.30.9.21 netmask 0xfff0 broadcast 172.30.9.31
Now to the strange part. I see that the selected route in R1 points to FW1
even though carp20/21 on FW1 is in state BACKUP. No matter what I do, apart
from setting static metrics, ospfd on R1 always selects FW1 as nexthop.
root@FW1:~ # ifconfig vlan21 | grep inet
inet 172.30.9.34 netmask 0xfff0 broadcast 172.30.9.47
root@FW1:~ # ifconfig carp20 | grep carp:
carp: BACKUP carpdev vlan20 vhid 1 advbase 1 advskew 10
root@FW1:~ # ifconfig carp21 | grep carp:
carp: BACKUP carpdev vlan21 vhid 1 advbase 1 advskew 10
root@FW2:~ # ifconfig vlan21 | grep inet
inet 172.30.9.35 netmask 0xfff0 broadcast 172.30.9.47
root@FW2:~ # ifconfig carp20 | grep carp:
carp: MASTER carpdev vlan20 vhid 1 advbase 1 advskew 100
root@FW2:~ # ifconfig carp21 | grep carp:
carp: MASTER carpdev vlan21 vhid 1 advbase 1 advskew 100
root@R1:~ # ospfctl sh
neighID Pri StateDeadTime Address Iface
Uptime
172.30.9.4 1 FULL/OTHER 00:00:38 172.30.9.35 vlan2100:21:33
172.30.9.3 1 FULL/BCKUP 00:00:38 172.30.9.34 vlan2100:22:14
root@R1:~ # ospfctl sh fib | grep 172.30.9.16/2
*O 32 172.30.9.16/28 172.30.9.34
*O 32 172.30.9.16/28 172.30.9.35
root@R1:~ # ospfctl sh rib | grep 172.30.9.16/2
172.30.9.16/28 172.30.9.34 Intra-Area Network 20
00:30:33
172.30.9.16/28 172.30.9.35 Intra-Area Network 20
00:29:56
root@R1:~ # route -n get 172.30.9.21
route to: 172.30.9.21
destination: 172.30.9.16
mask: 255.255.255.240
gateway: 172.30.9.34
interface: vlan21
if address: 172.30.9.37
priority: 32 (ospf)
flags:
use mtuexpire
11 0 0
As seen above R1 selects 172.30.9.34 as the nexthop based on ospf which is
wrong. It should be 172.30.9.35 as FW2 is carp master for carp20/21. What I
in the end want to achieve is that the router with carp20/21 MASTER should
be the preferred carp20 nexthop. An assumption can be made that carp20/21
will always have the same FW as master in my case.
iked vs IPsec failover (carp & sasyncd)
Hi folks, wrt IPsec failover via sasyncd and carp: sasyncd(8) and iked(8) don't seem to tell, but I would guess that all hosts on the carp interface have to share the private key to support renegotiation. How can I tell iked which private key to use, instead of local.key? Is there a similar naming scheme as for the foreign public keys? Every insightful comment is highly appreciated Harri
Re: CARP load balancing problems under KVM
On 21 Oct 07:12, Carlos Lopez wrote: > Hi all, > > Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was using > carp in IP balance mode without problems from several months. These firewalls > are installed in a RHEL 8.2 (fully patched) KVM host. > > After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I have > tested reconfiguring balance mode for ip-stealth and ip-unicast also and the > result is always the same: network packets are not processed by firewalls. > But if I configure CARP using “the simple configuration” and one node is > master and the other is backup all it is working without problems. > > Since the configuration is exactly the same for version 6.8 as it was for > version 6.7, could there be some kind of problem with the virtio driver on > the network interfaces in version 6.8? > > Regards, > C. L. Martinez Hi folks, I run into exactly the same issue with my ldap proxy. When one node is master for both carp nodes it responds to packets addressed to the carp ip. In load balancing mode it sometimes responds depending on the source ip. Please open a bug report. -- wq: ~uw
Re: CARP load balancing problems under KVM
Ok, done. I have already sent the bug report. On 21/10/2020, 11:11, "Uwe Werler" wrote: On 21 Oct 07:12, Carlos Lopez wrote: > Hi all, > > Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was using carp in IP balance mode without problems from several months. These firewalls are installed in a RHEL 8.2 (fully patched) KVM host. > > After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I have tested reconfiguring balance mode for ip-stealth and ip-unicast also and the result is always the same: network packets are not processed by firewalls. But if I configure CARP using “the simple configuration” and one node is master and the other is backup all it is working without problems. > > Since the configuration is exactly the same for version 6.8 as it was for version 6.7, could there be some kind of problem with the virtio driver on the network interfaces in version 6.8? > > Regards, > C. L. Martinez Hi folks, I run into exactly the same issue with my ldap proxy. When one node is master for both carp nodes it responds to packets addressed to the carp ip. In load balancing mode it sometimes responds depending on the source ip. Please open a bug report. -- wq: ~uw
CARP load balancing problems under KVM
Hi all, Before upgrade from OpenBSD 6.7 to OpenBSD 6.8, my pair firewalls was using carp in IP balance mode without problems from several months. These firewalls are installed in a RHEL 8.2 (fully patched) KVM host. After upgrading to OpenBSD 6.8, carp ip balance mode doesn’t works. I have tested reconfiguring balance mode for ip-stealth and ip-unicast also and the result is always the same: network packets are not processed by firewalls. But if I configure CARP using “the simple configuration” and one node is master and the other is backup all it is working without problems. Since the configuration is exactly the same for version 6.8 as it was for version 6.7, could there be some kind of problem with the virtio driver on the network interfaces in version 6.8? Regards, C. L. Martinez
Re: dhclient on carp
> On 23 Jul 2020, at 22:28, Guy Godfroy wrote: > > Doesn't work better. > I guess Sebastian is right, carp has to be assigned an IP to come up. yeah, i just read the code a bit. they have to be able to communicate to be able to elect which one is the active and which is the backup. i suggest using an address like one in 169.254.x.y/16 so the carps can elect. > > Le 23/07/2020 à 03:15, David Gwynne a écrit : >>> On 22 Jul 2020, at 22:59, Guy Godfroy wrote: >>> >>> Hello, >>> >>> So I read in 6.7 release note that it's finally possible to use dhclient on >>> CARP interface. That's great news. >>> >>> However, I'm not sure how to use it on a hostname.if file. I tried to >>> replace inet instruction directly with dhcp: >>> >>>dhcp vhid 11 carpdev em1 pass description "test" >>> >>> >>> But that didn't do the trick: at boot time, none of my nodes carp were in >>> master state so dhclient didn't manage to get any lease. >>> >>> So I have first to give a static IP to my carp in order to activate it, and >>> only then trigger dhcp: >>> >>>inet [...] vhid 11 carpdev em1 pass description "test" >>> >>>dhcp >>> >>> It doesn't feel right. Is there a better way to do this? >> hostname.if0 lines don't have to all be address configurations. generally >> netstart just passes the statements directly to ifconfig. >> does something like the following work in hostname.carp0? >> description "test" >> vhid 11 carpdev em1 pass >> dhcp >> dlg >
Re: dhclient on carp
Doesn't work better. I guess Sebastian is right, carp has to be assigned an IP to come up. Le 23/07/2020 à 03:15, David Gwynne a écrit : On 22 Jul 2020, at 22:59, Guy Godfroy wrote: Hello, So I read in 6.7 release note that it's finally possible to use dhclient on CARP interface. That's great news. However, I'm not sure how to use it on a hostname.if file. I tried to replace inet instruction directly with dhcp: dhcp vhid 11 carpdev em1 pass description "test" But that didn't do the trick: at boot time, none of my nodes carp were in master state so dhclient didn't manage to get any lease. So I have first to give a static IP to my carp in order to activate it, and only then trigger dhcp: inet [...] vhid 11 carpdev em1 pass description "test" dhcp It doesn't feel right. Is there a better way to do this? hostname.if0 lines don't have to all be address configurations. generally netstart just passes the statements directly to ifconfig. does something like the following work in hostname.carp0? description "test" vhid 11 carpdev em1 pass dhcp dlg
Re: dhclient on carp
> On 22 Jul 2020, at 22:59, Guy Godfroy wrote: > > Hello, > > So I read in 6.7 release note that it's finally possible to use dhclient on > CARP interface. That's great news. > > However, I'm not sure how to use it on a hostname.if file. I tried to replace > inet instruction directly with dhcp: > >dhcp vhid 11 carpdev em1 pass description "test" > > > But that didn't do the trick: at boot time, none of my nodes carp were in > master state so dhclient didn't manage to get any lease. > > So I have first to give a static IP to my carp in order to activate it, and > only then trigger dhcp: > >inet [...] vhid 11 carpdev em1 pass description "test" > >dhcp > > It doesn't feel right. Is there a better way to do this? hostname.if0 lines don't have to all be address configurations. generally netstart just passes the statements directly to ifconfig. does something like the following work in hostname.carp0? description "test" vhid 11 carpdev em1 pass dhcp dlg
