Re: CARP interfaces and mastership issue
Op 29-09-11 16:49, Damien Fleuriot schreef:
Quick follow-up again.
This is the code for sys/netinet/ip_carp.c on FreeBSD 8.2, OpenBSD
3.8, OpenBSD 3.9 in function carp_setrun(struct carp_softc *sc,
sa_family_t af)
FREEBSD 8.2-PRERELEASE with init + preempt = auto MASTER bug
Function starts at line 1371.
---
switch (sc-sc_state) {
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_send_ad_locked(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
CARP_LOG(%s: INIT - MASTER (preempting)\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
} else {
CARP_LOG(%s: INIT - BACKUP\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.8 with init + preempt = auto MASTER bug
Function starts at line 1293.
---
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
carp_send_ad(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
} else {
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.9 with bug fixed
Function starts at line 1348.
---
switch (sc-sc_state) {
case INIT:
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
break;
---
It looks like the root cause is there.
I'll rebuild and test, keep you updated.
Find below my test results with the OpenBSD39 implementation which
forces an INIT - BACKUP transition regardless of preempt.
# sysctl net.inet.carp.preempt
net.inet.carp.preempt: 1
# sysctl net.inet.carp.suppress_preempt
net.inet.carp.suppress_preempt: 0
# ifconfig carp17
carp17: flags=8LOOPBACK metric 0 mtu 1500
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
carp: INIT vhid 117 advbase 1 advskew 200
# ifconfig carp17 up; ./check_carp17_status.sh
count: 0carp: BACKUP vhid 117 advbase 1 advskew 200
count: 1carp: BACKUP vhid 117 advbase 1 advskew 200
count: 2carp: BACKUP vhid 117 advbase 1 advskew 200
count: 3carp: BACKUP vhid 117 advbase 1 advskew 200
count: 4carp: BACKUP vhid 117 advbase 1 advskew 200
count: 5carp: BACKUP vhid 117 advbase 1 advskew 200
count: 6carp: BACKUP vhid 117 advbase 1 advskew 200
count: 7carp: BACKUP vhid 117 advbase 1 advskew 200
count: 8carp: BACKUP vhid 117 advbase 1 advskew 200
count: 9carp: BACKUP vhid 117 advbase 1 advskew 200
count: 10 carp: BACKUP vhid 117 advbase 1 advskew 200
# dmesg
carp17: INIT - BACKUP
carp17: link state changed to DOWN
Looks like it works.
I'm afraid I cannot test actual preemption by shutting down a physical
interface on the MASTER, because they're actually used in production and
I've got users logged in to their VPN on pf1.
I see no reason this should break anything however, it only forces the
CARP interface to assume a BACKUP state right after INIT, as it normally
should, regardless of preemption.
I'm filling a PR + patch.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Do you have the PR number for me and the patch, i have the same problem
and like it to be fixed.
regards
Johan Hendriks
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
On 1 October 2011 11:04, johan Hendriks joh.hendr...@gmail.com wrote:
Op 29-09-11 16:49, Damien Fleuriot schreef:
Quick follow-up again.
This is the code for sys/netinet/ip_carp.c on FreeBSD 8.2, OpenBSD
3.8, OpenBSD 3.9 in function carp_setrun(struct carp_softc *sc,
sa_family_t af)
FREEBSD 8.2-PRERELEASE with init + preempt = auto MASTER bug
Function starts at line 1371.
---
switch (sc-sc_state) {
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt)
{
carp_send_ad_locked(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
CARP_LOG(%s: INIT - MASTER (preempting)\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
} else {
CARP_LOG(%s: INIT - BACKUP\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.8 with init + preempt = auto MASTER bug
Function starts at line 1293.
---
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt)
{
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
carp_send_ad(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
} else {
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.9 with bug fixed
Function starts at line 1348.
---
switch (sc-sc_state) {
case INIT:
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
break;
---
It looks like the root cause is there.
I'll rebuild and test, keep you updated.
Find below my test results with the OpenBSD39 implementation which
forces an INIT - BACKUP transition regardless of preempt.
# sysctl net.inet.carp.preempt
net.inet.carp.preempt: 1
# sysctl net.inet.carp.suppress_preempt
net.inet.carp.suppress_preempt: 0
# ifconfig carp17
carp17: flags=8LOOPBACK metric 0 mtu 1500
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
carp: INIT vhid 117 advbase 1 advskew 200
# ifconfig carp17 up; ./check_carp17_status.sh
count: 0 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 1 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 2 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 3 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 4 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 5 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 6 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 7 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 8 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 9 carp: BACKUP vhid 117 advbase 1 advskew 200
count: 10 carp: BACKUP vhid 117 advbase 1 advskew 200
# dmesg
carp17: INIT - BACKUP
carp17: link state changed to DOWN
Looks like it works.
I'm afraid I cannot test actual preemption by shutting down a physical
interface on the MASTER, because they're actually used in production and
I've got users logged in to their VPN on pf1.
I see no reason this should break anything however, it only forces the
CARP interface to assume a BACKUP state right after INIT, as it normally
should, regardless of preemption.
I'm filling a PR + patch.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Do you have the PR number for me and the patch, i have the same problem and
like it to be fixed.
regards
Johan Hendriks
http://www.freebsd.org/cgi/query-pr.cgi?pr=161123
Here you go.
Please note I do not have extensive C programming experience.
I have simply removed the part that makes the CARP interface
immediately transition from INIT to MASTER when preemption is enabled,
and replaced it with an INIT-BACKUP transition like in OpenBSD3.9 in
which the bug is fixed.
I cannot guarantee the absence of side effects, although I couldn't
find any in my testing.
Also note that if you update your sources the patch will obviously be
gone, take care when
Re: CARP interfaces and mastership issue
On 9/15/11 11:07 AM, Damien FLEURIOT wrote: Hello list, TLDR: carp interface becomes MASTER for a split second after being created, even if another MASTER exists on the network with faster advertisements. Breaks connections. HOWTO prevent ? We've been experiencing this double mastership problem with CARP interfaces. Allow me to put some context here: 2 firewalls, PF1, PF2, each with 2 VLANs (for example, some have more) on a lagg device (link aggregation). These firewalls then share virtual IPs through CARP interfaces, let us assume the following: PF1: - vlan13 - vlan410 - carp13 (advskew 50) - carp410 (advskew 50) PF2: - vlan13 - vlan410 - carp13 (advskew 100) - carp410 (advskew 100) CARP preemption is turned on, so that if vlan13 should fail on PF1, PF2 would assume mastership on both CARP interfaces. Syscontrols below: net.inet.carp.allow: 1 net.inet.carp.preempt: 1 net.inet.carp.log: 1 net.inet.carp.arpbalance: 0 net.inet.carp.suppress_preempt: 0 The problem we have is, say for example we reboot PF2. When it comes back up, it will, even for a split second, assume CARP mastership for its interfaces, at the same time as PF1. This breaks existing sessions, openvpn tunnels and new client connections. While I acknowledge the home-made demons should be built to support tiny network outages, this doesn't solve our main problem. We have the same issue when destroying/creating said CARP interfaces. Recently we upgraded some switches' IOS version on our backup datacenter (which also has 2 PF boxes, sharing the CARP IPs with the 2 PFs on our production DC). To prevent anything nasty happening, we forbade production VLANs on the switches' uplink ports and only allowed management traffic to allow us to perform the upgrade. Things went smoothly but when we brought the production VLANs up again at layer 2 on the switches, when spanning-tree converged we had again a double MASTER problem. I understand I could have avoided it by destroying/recreating the CARP interfaces, but even in this case there is a split second during which both firewalls are CARP MASTER. Is there any way to force CARP to assume INIT state for some time when coming up, and only after X seconds either become MASTER or BACKUP ? Any other idea how to solve this, guys ? ___ freebsd...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org Hello List, This is a follow-up to my original email quoted above. It seems that there is an existing bug in OpenBSD 3.8 and lower's CARP implementation which causes CARP interfaces to skip the INIT state altogether and start as MASTER if preempt is enabled. Source: https://calomel.org/pf_carp.html Quote: INIT : All CARP interfaces start in this state. Also, when a CARP interface is admin down, i.e. ifconfig em0 down, it is put into this state. When a CARP interface is admin up, it immediately transitions to BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will cause the host to transition to MASTER right away if preempt is enabled. I have been able to verify and reproduce this behavior on boxes running both 8.1 and 8.2 FreeBSD. Does anyone know what version of OpenBSD's CARP implementation we're running on FreeBSD 8.x ? It seems like this is the same bug, to me. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
On 29 September 2011 14:20, Damien Fleuriot m...@my.gd wrote:
On 9/15/11 11:07 AM, Damien FLEURIOT wrote:
Hello list,
TLDR: carp interface becomes MASTER for a split second after being
created, even if another MASTER exists on the network with faster
advertisements. Breaks connections. HOWTO prevent ?
We've been experiencing this double mastership problem with CARP interfaces.
Allow me to put some context here:
2 firewalls, PF1, PF2, each with 2 VLANs (for example, some have more)
on a lagg device (link aggregation).
These firewalls then share virtual IPs through CARP interfaces, let us
assume the following:
PF1:
- vlan13
- vlan410
- carp13 (advskew 50)
- carp410 (advskew 50)
PF2:
- vlan13
- vlan410
- carp13 (advskew 100)
- carp410 (advskew 100)
CARP preemption is turned on, so that if vlan13 should fail on PF1, PF2
would assume mastership on both CARP interfaces.
Syscontrols below:
net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.log: 1
net.inet.carp.arpbalance: 0
net.inet.carp.suppress_preempt: 0
The problem we have is, say for example we reboot PF2.
When it comes back up, it will, even for a split second, assume CARP
mastership for its interfaces, at the same time as PF1.
This breaks existing sessions, openvpn tunnels and new client connections.
While I acknowledge the home-made demons should be built to support tiny
network outages, this doesn't solve our main problem.
We have the same issue when destroying/creating said CARP interfaces.
Recently we upgraded some switches' IOS version on our backup datacenter
(which also has 2 PF boxes, sharing the CARP IPs with the 2 PFs on our
production DC).
To prevent anything nasty happening, we forbade production VLANs on the
switches' uplink ports and only allowed management traffic to allow us
to perform the upgrade.
Things went smoothly but when we brought the production VLANs up again
at layer 2 on the switches, when spanning-tree converged we had again a
double MASTER problem.
I understand I could have avoided it by destroying/recreating the CARP
interfaces, but even in this case there is a split second during which
both firewalls are CARP MASTER.
Is there any way to force CARP to assume INIT state for some time when
coming up, and only after X seconds either become MASTER or BACKUP ?
Any other idea how to solve this, guys ?
___
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-pf
To unsubscribe, send any mail to freebsd-pf-unsubscr...@freebsd.org
Hello List,
This is a follow-up to my original email quoted above.
It seems that there is an existing bug in OpenBSD 3.8 and lower's CARP
implementation which causes CARP interfaces to skip the INIT state
altogether and start as MASTER if preempt is enabled.
Source:
https://calomel.org/pf_carp.html
Quote:
INIT : All CARP interfaces start in this state. Also, when a CARP
interface is admin down, i.e. ifconfig em0 down, it is put into this
state. When a CARP interface is admin up, it immediately transitions to
BACKUP. Note that in OpenBSD 3.8 and earlier, a bug exists which will
cause the host to transition to MASTER right away if preempt is enabled.
I have been able to verify and reproduce this behavior on boxes running
both 8.1 and 8.2 FreeBSD.
Does anyone know what version of OpenBSD's CARP implementation we're
running on FreeBSD 8.x ?
It seems like this is the same bug, to me.
Quick follow-up again.
This is the code for sys/netinet/ip_carp.c on FreeBSD 8.2, OpenBSD
3.8, OpenBSD 3.9 in function carp_setrun(struct carp_softc *sc,
sa_family_t af)
FREEBSD 8.2-PRERELEASE with init + preempt = auto MASTER bug
Function starts at line 1371.
---
switch (sc-sc_state) {
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_send_ad_locked(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
CARP_LOG(%s: INIT - MASTER (preempting)\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
} else {
CARP_LOG(%s: INIT - BACKUP\n, SC2IFP(sc)-if_xname);
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.8 with init + preempt = auto MASTER bug
Function starts at line 1293.
---
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
carp_send_ad(sc);
Re: CARP interfaces and mastership issue
Quick follow-up again.
This is the code for sys/netinet/ip_carp.c on FreeBSD 8.2, OpenBSD
3.8, OpenBSD 3.9 in function carp_setrun(struct carp_softc *sc,
sa_family_t af)
FREEBSD 8.2-PRERELEASE with init + preempt = auto MASTER bug
Function starts at line 1371.
---
switch (sc-sc_state) {
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_send_ad_locked(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
CARP_LOG(%s: INIT - MASTER (preempting)\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
} else {
CARP_LOG(%s: INIT - BACKUP\n,
SC2IFP(sc)-if_xname);
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.8 with init + preempt = auto MASTER bug
Function starts at line 1293.
---
case INIT:
if (carp_opts[CARPCTL_PREEMPT] !carp_suppress_preempt) {
carp_set_state(sc, MASTER);
carp_setroute(sc, RTM_ADD);
carp_send_ad(sc);
carp_send_arp(sc);
#ifdef INET6
carp_send_na(sc);
#endif /* INET6 */
} else {
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
}
break;
---
OPENBSD 3.9 with bug fixed
Function starts at line 1348.
---
switch (sc-sc_state) {
case INIT:
carp_set_state(sc, BACKUP);
carp_setroute(sc, RTM_DELETE);
carp_setrun(sc, 0);
break;
---
It looks like the root cause is there.
I'll rebuild and test, keep you updated.
Find below my test results with the OpenBSD39 implementation which
forces an INIT - BACKUP transition regardless of preempt.
# sysctl net.inet.carp.preempt
net.inet.carp.preempt: 1
# sysctl net.inet.carp.suppress_preempt
net.inet.carp.suppress_preempt: 0
# ifconfig carp17
carp17: flags=8LOOPBACK metric 0 mtu 1500
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
inet 46.182.[snip] netmask 0x
carp: INIT vhid 117 advbase 1 advskew 200
# ifconfig carp17 up; ./check_carp17_status.sh
count: 0carp: BACKUP vhid 117 advbase 1 advskew 200
count: 1carp: BACKUP vhid 117 advbase 1 advskew 200
count: 2carp: BACKUP vhid 117 advbase 1 advskew 200
count: 3carp: BACKUP vhid 117 advbase 1 advskew 200
count: 4carp: BACKUP vhid 117 advbase 1 advskew 200
count: 5carp: BACKUP vhid 117 advbase 1 advskew 200
count: 6carp: BACKUP vhid 117 advbase 1 advskew 200
count: 7carp: BACKUP vhid 117 advbase 1 advskew 200
count: 8carp: BACKUP vhid 117 advbase 1 advskew 200
count: 9carp: BACKUP vhid 117 advbase 1 advskew 200
count: 10 carp: BACKUP vhid 117 advbase 1 advskew 200
# dmesg
carp17: INIT - BACKUP
carp17: link state changed to DOWN
Looks like it works.
I'm afraid I cannot test actual preemption by shutting down a physical
interface on the MASTER, because they're actually used in production and
I've got users logged in to their VPN on pf1.
I see no reason this should break anything however, it only forces the
CARP interface to assume a BACKUP state right after INIT, as it normally
should, regardless of preemption.
I'm filling a PR + patch.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
Le Sat, 17 Sep 2011 23:40:06 -0400 (EDT), Brian Seklecki (Mobile) laval...@probikesllc.com a écrit : What would help here, is for a carp interface to wait a given delay (tunable through a sysctl ?) after creation or after being brought up I see now. The tunable sounds like a good idea; we should check OpenBSD, they probably already implemented something and we're behind. If not, a preempt dampener feature would be an awesome return feature. Might need to implment another state: MASTER-LISTENING (or LEARNING) ah a STP. OpenBSD uses a carp demote counter that prevents to become master (but it will become master anyway if there is not carp advertizement on the interface). There is a sysctl in FreeBSD but it's readonly. This is used to delay carp until pfsync synchronization is done and by daemons like bgpd. Anyway if carp becomes master when the interface is set up, it looks to be a bug on FreeBSD (and if you are sure that the problem does not come from the switch). That works fine on OpenBSD. Regards. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
On 18 Sep 2011, at 12:44, Patrick Lamaiziere patf...@davenulle.org wrote: Le Sat, 17 Sep 2011 23:40:06 -0400 (EDT), Brian Seklecki (Mobile) laval...@probikesllc.com a écrit : What would help here, is for a carp interface to wait a given delay (tunable through a sysctl ?) after creation or after being brought up I see now. The tunable sounds like a good idea; we should check OpenBSD, they probably already implemented something and we're behind. If not, a preempt dampener feature would be an awesome return feature. Might need to implment another state: MASTER-LISTENING (or LEARNING) ah a STP. OpenBSD uses a carp demote counter that prevents to become master (but it will become master anyway if there is not carp advertizement on the interface). There is a sysctl in FreeBSD but it's readonly. This is used to delay carp until pfsync synchronization is done and by daemons like bgpd. Anyway if carp becomes master when the interface is set up, it looks to be a bug on FreeBSD (and if you are sure that the problem does not come from the switch). This can be easily verified. When our vlan13 is forwarding on the switch, destroy and recreate the carp13 interface. It still assumes mastership during a short time, then yields and becomes backup.___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
On Sep 15, 2011, at 23:14 , Damien Fleuriot wrote: What would help here, is for a carp interface to wait a given delay (tunable through a sysctl ?) after creation or after being brought up from down. I have the same observation. Perhaps it can just avoid going up initially --- it will become master anyway if it does not hear anything on the interface. Daniel___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
What would help here, is for a carp interface to wait a given delay (tunable through a sysctl ?) after creation or after being brought up I see now. The tunable sounds like a good idea; we should check OpenBSD, they probably already implemented something and we're behind. If not, a preempt dampener feature would be an awesome return feature. Might need to implment another state: MASTER-LISTENING (or LEARNING) ah a STP. ~BAS I have the same observation. Perhaps it can just avoid going up initially --- it will become master anyway if it does not hear anything on the interface. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
CARP interfaces and mastership issue
Posting to -stable since -pf didn't show much interest :/ Original Message Subject: CARP interfaces and mastership issue Date: Thu, 15 Sep 2011 11:07:37 +0200 Hello list, TLDR: carp interface becomes MASTER for a split second after being created, even if another MASTER exists on the network with faster advertisements. Breaks connections. HOWTO prevent ? We've been experiencing this double mastership problem with CARP interfaces. Allow me to put some context here: 2 firewalls, PF1, PF2, each with 2 VLANs (for example, some have more) on a lagg device (link aggregation). These firewalls then share virtual IPs through CARP interfaces, let us assume the following: PF1: - vlan13 - vlan410 - carp13 (advskew 50) - carp410 (advskew 50) PF2: - vlan13 - vlan410 - carp13 (advskew 100) - carp410 (advskew 100) CARP preemption is turned on, so that if vlan13 should fail on PF1, PF2 would assume mastership on both CARP interfaces. Syscontrols below: net.inet.carp.allow: 1 net.inet.carp.preempt: 1 net.inet.carp.log: 1 net.inet.carp.arpbalance: 0 net.inet.carp.suppress_preempt: 0 The problem we have is, say for example we reboot PF2. When it comes back up, it will, even for a split second, assume CARP mastership for its interfaces, at the same time as PF1. This breaks existing sessions, openvpn tunnels and new client connections. While I acknowledge the home-made demons should be built to support tiny network outages, this doesn't solve our main problem. We have the same issue when destroying/creating said CARP interfaces. Recently we upgraded some switches' IOS version on our backup datacenter (which also has 2 PF boxes, sharing the CARP IPs with the 2 PFs on our production DC). To prevent anything nasty happening, we forbade production VLANs on the switches' uplink ports and only allowed management traffic to allow us to perform the upgrade. Things went smoothly but when we brought the production VLANs up again at layer 2 on the switches, when spanning-tree converged we had again a double MASTER problem. I understand I could have avoided it by destroying/recreating the CARP interfaces, but even in this case there is a split second during which both firewalls are CARP MASTER. Is there any way to force CARP to assume INIT state for some time when coming up, and only after X seconds either become MASTER or BACKUP ? Any other idea how to solve this, guys ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
Things went smoothly but when we brought the production VLANs up again at layer 2 on the switches, when spanning-tree converged we had again a double MASTER problem. In older versions of FBSD, creating logical interfaces like vlan(4) and carp(4) had an nasty inadvertent side effect of toggling the state of the underlying phyiscal interface. This may be fixed in newer version. This would then then cause STP to reset on the switchport which can take up to 50 seconds to restore. In the mean time, the backup host hasn't heard from the master and assume the role of master. You can try turning on switchport spanning-tree portfast on your backup system which should cut down this time signifantly. If you can assure that no STP BPDUs will be announced from your CARP system, then its probably safe to run PortFast on a trunk. The same is true after a reboot. Maybe hack the RC script to force the CARP interfaces on your backup to stay down at boot time for an extra 10/15 seconds ~BAS I understand I could have avoided it by destroying/recreating the CARP interfaces, but even in this case there is a split second during which both firewalls are CARP MASTER. Is there any way to force CARP to assume INIT state for some time when coming up, and only after X seconds either become MASTER or BACKUP ? Any other idea how to solve this, guys ? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: CARP interfaces and mastership issue
On 15 September 2011 18:12, Brian Seklecki (Mobile) laval...@probikesllc.com wrote: Things went smoothly but when we brought the production VLANs up again at layer 2 on the switches, when spanning-tree converged we had again a double MASTER problem. In older versions of FBSD, creating logical interfaces like vlan(4) and carp(4) had an nasty inadvertent side effect of toggling the state of the underlying phyiscal interface. We're running quite the recent version really: FreeBSD 8.2-STABLE FreeBSD 8.2-STABLE #1: Thu Sep 1 15:51:49 CEST 2011 root@pf2-dmz:/usr/obj/usr/src/sys/FW amd64 This may be fixed in newer version. This would then then cause STP to reset on the switchport which can take up to 50 seconds to restore. Switchports are running portfast trunk with RPVST, this was a good candidate but sadly not the cause here. Also, physical interfaces do not get reset, dmesg is clean. In the mean time, the backup host hasn't heard from the master and assume the role of master. You can try turning on switchport spanning-tree portfast on your backup system which should cut down this time signifantly. If you can assure that no STP BPDUs will be announced from your CARP system, then its probably safe to run PortFast on a trunk. This is already done on all non-uplink ports, with portfast trunk as appropriate for servers that tag VLANs. The same is true after a reboot. Maybe hack the RC script to force the CARP interfaces on your backup to stay down at boot time for an extra 10/15 seconds Even then, once we bring the interface up (or create it), it seems to immediately assume mastership, breaking existing sessions on the main firewall, THEN it says it's received a more frequent advertisement and swaps to backup. What would help here, is for a carp interface to wait a given delay (tunable through a sysctl ?) after creation or after being brought up from down. This would allow the server to receive VRRP advertisements and immediately assume a BACKUP carp role upon boot or interface creation/up. This would also retain CARP's high reactivity to network failures and outages since this would only apply to a newly created/up'ed interface, not an already running one. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
