Re: carp backup becomes no carrier

Johan Huldtgren Fri, 24 Apr 2015 18:22:56 -0700

hello,

a few hours after I sent the previous e-mail the backup
(April 23rd snap) took over and became the master, at
that point I could not reach the carp interfaces anymore.
Reverting roles so the host running the April 12th snap
became the master would mostly fix the problems although
occasionally things would seem to get confused and traffic
(esp to vlan666, which my laptop isn't on but has access
to) would cease. Shutting down the node running the April
23rd snap would generally clear this up, but I'm not sure
if this is a red herring and there is some caching going
on somewhere which is clouding my troubleshooting efforts.


Regardless, I stood up an i386 vm, downloaded -current but
grabbed ip_carp.c r1.249, and built a new kernel. Copied it
over to the firewall which had the April 23rd snap and now
everything is working as it was before. Traffic is flowing
as expected regardless of which host is master and which is
backup. It's only been a few hours, but so far so good.

thanks,

.jh


On 2015-04-24 13:15, Johan Huldtgren wrote:

hello,

I noticed some carp weirdness and sthen@ thought it might be worth
bringing to light.  Quick background, I run two carp nodes, one
(current master) is running the April 12th snapshot, the other is
running the April 23rd snapshot. The node running the April 23rd
snap when it's the backup node ifconfig reports all the carp
interfaces status' as "no carrier" whereas before (as far as I can
remember and on the April 12th snap at least) it would report
"backup". Once the backup becomes the master status changes to
"master".

I don't notice anything not working, however this behavior is perhaps
not expected.

dmesgs and ifconfig output for each host below, let me know if you
need anything further.

thanks,

.jh

April 12th snapshot host:

$ dmesg
syncing disks... done
OpenBSD 5.7-current (GENERIC) #772: Sun Apr 12 17:38:03 MDT 2015
    dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD"
586-class) 500 MHz
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
real mem  = 536363008 (511MB)
avail mem = 515301376 (491MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 20/71/05, BIOS32 rev. 0 @ 0xfac40
pcibios0 at bios0: rev 2.0 @ 0xf0000/0x10000
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc8000/0xa800
cpu0 at mainbus0: (uniprocessor)
mtrr: K6-family MTRR support (2 registers)
amdmsr0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
0:20:0: io address conflict 0x6100/0x100
0:20:0: io address conflict 0x6200/0x200
pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x31
glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
vr0 at pci0 dev 6 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11,
address 00:00:24:c9:58:4c
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr1 at pci0 dev 7 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 5,
address 00:00:24:c9:58:4d
ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr2 at pci0 dev 8 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 9,
address 00:00:24:c9:58:4e
ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr3 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 12,
address 00:00:24:c9:58:4f
ukphy3 at vr3 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
ral0 at pci0 dev 17 function 0 "Ralink RT2561S" rev 0x00: irq 15,
address 00:12:0e:61:7f:b0
ral0: MAC/BBP RT2561C, RF RT5225
glxpcib0 at pci0 dev 20 function 0 "AMD CS5536 ISA" rev 0x03: rev 3,
32-bit 3579545Hz timer, watchdog, gpio, i2c
gpio0 at glxpcib0: 32 pins
iic0 at glxpcib0
pciide0 at pci0 dev 20 function 2 "AMD CS5536 IDE" rev 0x01: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFX4-8192>
wd0: 4-sector PIO, LBA, 7815MB, 16007040 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
ohci0 at pci0 dev 21 function 0 "AMD CS5536 USB" rev 0x02: irq 7,
version 1.0, legacy support
ehci0 at pci0 dev 21 function 1 "AMD CS5536 USB" rev 0x02: irq 7
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
isa0 at glxpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbc0: unable to establish interrupt for aux slot
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
nsclpcsio0 at isa0 port 0x2e/2: NSC PC87366 rev 10: GPIO VLM TMS
gpio1 at nsclpcsio0: 29 pins
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "AMD OHCI root hub" rev 1.00/1.00 addr 1
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (ba730608caf94ae4.a) swap on wd0b dump on wd0b
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp0: state transition: MASTER -> BACKUP
carp1: state transition: MASTER -> BACKUP
carp2: state transition: MASTER -> BACKUP
carp3: state transition: MASTER -> BACKUP
carp3: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp0: state transition: BACKUP -> MASTER

$ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
    priority: 0
    groups: lo
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
    inet6 ::1 prefixlen 128
    inet 127.0.0.1 netmask 0xff000000
vr0:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c9:58:4c
    priority: 0
    groups: egress
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
    inet 172.16.0.2 netmask 0xffffff00 broadcast 172.16.0.255
vr1:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c9:58:4d
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
vr2:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c9:58:4e
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
vr3:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c9:58:4f
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
enc0: flags=0<>
    priority: 0
    groups: enc
    status: active
ral0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:12:0e:61:7f:b0
    priority: 4
    groups: wlan
    media: IEEE802.11 autoselect
    status: no network
    ieee80211: nwid "" 100dBm

vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c9:58:4d
    priority: 0
    vlan: 20 parent interface: vr1
    groups: vlan
    status: active
    inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255

vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c9:58:4e
    priority: 0
    vlan: 30 parent interface: vr2
    groups: vlan
    status: active
    inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255

vlan666: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c9:58:4f
    priority: 0
    vlan: 666 parent interface: vr3
    groups: vlan
    status: active
    inet 10.66.66.2 netmask 0xffffff00 broadcast 10.66.66.255
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
    priority: 0
    groups: tun
    status: active
    inet 10.6.6.1 --> 10.6.6.2 netmask 0xffffffff
pfsync0: flags=41<UP,RUNNING> mtu 1500
    priority: 0

pfsync: syncdev: vlan666 syncpeer: 10.66.66.3 maxupd: 128 defer:off

    groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33192
    priority: 0
    groups: pflog
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:01
    priority: 0

carp: MASTER carpdev vr0 vhid 1 advbase 1 advskew 100 carppeer172.16.0.3

    groups: carp
    status: master
    inet 172.16.0.103 netmask 0xffffff00 broadcast 172.16.0.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:02
    priority: 0
    carp: MASTER carpdev vlan20 vhid 2 advbase 1 advskew 100 carppeer
192.168.100.3
    groups: carp
    status: master
    inet 192.168.100.103 netmask 0xffffff00 broadcast 192.168.100.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:03
    priority: 0
    carp: MASTER carpdev vlan30 vhid 3 advbase 1 advskew 100 carppeer
192.168.0.3
    groups: carp
    status: master
    inet 192.168.0.103 netmask 0xffffff00 broadcast 192.168.0.255
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:04
    priority: 0
    carp: MASTER carpdev vlan666 vhid 4 advbase 1 advskew 100 carppeer
10.66.66.3
    groups: carp
    status: master
    inet 10.66.66.103 netmask 0xffffff00 broadcast 10.66.66.255


April 23rd snapshot host:

$ dmesg
syncing disks... done
OpenBSD 5.7-current (GENERIC) #806: Thu Apr 23 23:54:49 MDT 2015
    dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD"
586-class) 500 MHz
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
real mem  = 536363008 (511MB)
avail mem = 515309568 (491MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 20/80/26, BIOS32 rev. 0 @ 0xfac40
pcibios0 at bios0: rev 2.0 @ 0xf0000/0x10000
pcibios0: pcibios_get_intr_routing - function not supported
pcibios0: PCI IRQ Routing information unavailable.
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc8000/0xa800
cpu0 at mainbus0: (uniprocessor)
mtrr: K6-family MTRR support (2 registers)
amdmsr0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
0:20:0: io address conflict 0x6100/0x100
0:20:0: io address conflict 0x6200/0x200
pchb0 at pci0 dev 1 function 0 "AMD Geode LX" rev 0x30
glxsb0 at pci0 dev 1 function 2 "AMD Geode LX Crypto" rev 0x00: RNG AES
vr0 at pci0 dev 6 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 11,
address 00:00:24:c8:da:54
ukphy0 at vr0 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr1 at pci0 dev 7 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 5,
address 00:00:24:c8:da:55
ukphy1 at vr1 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr2 at pci0 dev 8 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 9,
address 00:00:24:c8:da:56
ukphy2 at vr2 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
vr3 at pci0 dev 9 function 0 "VIA VT6105M RhineIII" rev 0x96: irq 12,
address 00:00:24:c8:da:57
ukphy3 at vr3 phy 1: Generic IEEE 802.3u media interface, rev. 3: OUI
0x004063, model 0x0034
ipw0 at pci0 dev 17 function 0 "Intel PRO/Wireless 2100" rev 0x04: irq
15, address 00:04:23:83:7a:a1
glxpcib0 at pci0 dev 20 function 0 "AMD CS5536 ISA" rev 0x03: rev 3,
32-bit 3579545Hz timer, watchdog, gpio, i2c
gpio0 at glxpcib0: 32 pins
iic0 at glxpcib0
pciide0 at pci0 dev 20 function 2 "AMD CS5536 IDE" rev 0x01: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFX4-8192>
wd0: 4-sector PIO, LBA, 7815MB, 16007040 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
ohci0 at pci0 dev 21 function 0 "AMD CS5536 USB" rev 0x02: irq 7,
version 1.0, legacy support
ehci0 at pci0 dev 21 function 1 "AMD CS5536 USB" rev 0x02: irq 7
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
isa0 at glxpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbc0: unable to establish interrupt for aux slot
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
nsclpcsio0 at isa0 port 0x2e/2: NSC PC87366 rev 9: GPIO VLM TMS
gpio1 at nsclpcsio0: 29 pins
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "AMD OHCI root hub" rev 1.00/1.00 addr 1
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on wd0a (a9a7a41b4f8b8947.a) swap on wd0b dump on wd0b
carp0: state transition: BACKUP -> MASTER
carp1: state transition: BACKUP -> MASTER
carp2: state transition: BACKUP -> MASTER
carp3: state transition: BACKUP -> MASTER
carp0: state transition: MASTER -> BACKUP
carp3: state transition: MASTER -> BACKUP
carp2: state transition: MASTER -> BACKUP
carp1: state transition: MASTER -> BACKUP
pfsync: failed to receive bulk update

$ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
    priority: 0
    groups: lo
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    inet6 ::1 prefixlen 128
    inet 127.0.0.1 netmask 0xff000000
vr0:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c8:da:54
    priority: 0
    groups: egress
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
    inet 172.16.0.3 netmask 0xffffff00 broadcast 172.16.0.255
vr1:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c8:da:55
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
vr2:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c8:da:56
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
vr3:
flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST>
mtu 1500
    lladdr 00:00:24:c8:da:57
    priority: 0
    media: Ethernet autoselect (100baseTX full-duplex)
    status: active
ipw0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:04:23:83:7a:a1
    priority: 4
    groups: wlan
    media: IEEE802.11 autoselect (autoselect mode 11b)
    status: no network
    ieee80211: nwid "" 100dBm
enc0: flags=0<>
    priority: 0
    groups: enc
    status: active

vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c8:da:55
    priority: 0
    vlan: 20 parent interface: vr1
    groups: vlan
    status: active
    inet 192.168.100.3 netmask 0xffffff00 broadcast 192.168.100.255

vlan30: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c8:da:56
    priority: 0
    vlan: 30 parent interface: vr2
    groups: vlan
    status: active
    inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255

vlan666: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu1500

    lladdr 00:00:24:c8:da:57
    priority: 0
    vlan: 666 parent interface: vr3
    groups: vlan
    status: active
    inet 10.66.66.3 netmask 0xffffff00 broadcast 10.66.66.255
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
    priority: 0
    groups: tun
    status: active
    inet 10.6.6.1 --> 10.6.6.2 netmask 0xffffffff
pfsync0: flags=41<UP,RUNNING> mtu 1500
    priority: 0

pfsync: syncdev: vlan666 syncpeer: 10.66.66.2 maxupd: 128 defer:off

    groups: carp pfsync
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33192
    priority: 0
    groups: pflog
carp3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:04
    priority: 0

carp: BACKUP carpdev vlan666 vhid 4 advbase 1 advskew 0 carppeer10.66.66.2

    groups: carp
    status: no carrier
    inet 10.66.66.103 netmask 0xffffff00 broadcast 10.66.66.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:03
    priority: 0

carp: BACKUP carpdev vlan30 vhid 3 advbase 1 advskew 0 carppeer192.168.0.2

    groups: carp
    status: no carrier
    inet 192.168.0.103 netmask 0xffffff00 broadcast 192.168.0.255
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:02
    priority: 0
    carp: BACKUP carpdev vlan20 vhid 2 advbase 1 advskew 0 carppeer
192.168.100.2
    groups: carp
    status: no carrier
    inet 192.168.100.103 netmask 0xffffff00 broadcast 192.168.100.255
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    lladdr 00:00:5e:00:01:01
    priority: 0

carp: BACKUP carpdev vr0 vhid 1 advbase 1 advskew 0 carppeer172.16.0.2

    groups: carp
    status: no carrier
    inet 172.16.0.103 netmask 0xffffff00 broadcast 172.16.0.255

Re: carp backup becomes no carrier

Reply via email to