Hi all,
I set up FW1 and FW2 with trunk and CARP - If I set up the trunk
interface using trunkproto lacp and also specify a carppeer FW1 carp
interface does not return to master upon reboot and FW1 has 'carp
demote count 1'. I tried with various switch settings (lacp/active,
passive), em and bge interfaces, and the hosts plugged directly into
each other(no other switches to test with). I could bring everything
back to normal by doing carpdemote 100 on FW2.
I finally starting mucking around with the start up scripts and this
ineloquently 'fixes' the issue for me.
# diff -u -b /etc/netstart.orig /etc/netstart
--- /etc/netstart.orig Thu Nov 10 12:39:22 2011
+++ /etc/netstart Thu Nov 10 13:31:02 2011
@@ -263,7 +263,9 @@
# The trunk interfaces need to come up first in this list.
# The (s)vlan interfaces need to come up after trunk.
# Configure all the carp interfaces which we know about before default route.
-ifmstart "trunk svlan vlan carp"
+ifmstart "trunk"
+sleep 10
+ifmstart "svlan vlan carp"
if [ "$ip6kernel" = "YES" -a "x$rtsolif" != "x" ]; then
fw=`sysctl -n net.inet6.ip6.forwarding`
Unfortunately, the switch does not have anything close to 'show lacp
neighbor(detail)'. Just if the ports are up, lacp/static and
active/passive.
So is this my gear, (and/or) my setup, or does it happen to anyone else?
After reboot on FW1 logs -
FW1# tail -n 14 /var/log/messages
Nov 9 15:03:17 xcore1 /bsd: root on wd0a (deb23c6900dddd8c.a) swap on
wd0b dump on wd0b
Nov 9 15:03:17 xcore1 /bsd: carp1: state transition: INIT -> BACKUP
Nov 9 15:03:17 xcore1 /bsd: carp: pfsync0 demoted group carp by 1 to
129 (pfsync bulk start)
Nov 9 15:03:17 xcore1 /bsd: carp: pfsync0 demoted group pfsync by 1
to 1 (pfsync bulk start)
Nov 9 15:03:17 xcore1 /bsd: carp1: state transition: BACKUP -> MASTER
Nov 9 15:03:16 xcore1 savecore: no core dump
Nov 9 15:03:17 xcore1 /bsd: carp: pfsync0 demoted group carp by -1 to
128 (pfsync bulk done)
Nov 9 15:03:17 xcore1 /bsd: carp: pfsync0 demoted group pfsync by -1
to 0 (pfsync bulk done)
Nov 9 15:03:23 xcore1 /bsd: carp1: ip_output failed: 64
Nov 9 15:03:25 xcore1 last message repeated 3 times
Nov 9 15:03:25 xcore1 /bsd: carp: carp1 demoted group carp by 1 to 1
(> snderrors)
Nov 9 15:03:26 xcore1 /bsd: carp1: ip_output failed: 64
Nov 9 15:03:27 xcore1 /bsd: carp1: ip_output failed: 64
Nov 9 15:03:28 xcore1 /bsd: carp1: state transition: MASTER -> BACKUP
FW2#
Nov 9 15:02:18 xcore2 /bsd: carp1: state transition: BACKUP -> MASTER
FW1&FW2# tail -n 4 /etc/sysctl.conf
###additions
net.inet.carp.preempt=1
net.inet.ip.forwarding=1
net.inet.carp.log=7
FW1&FW2# pfctl -vvf /etc/pf.conf
set skip on { lo }
@0 pass all flags S/SA
@1 block drop in on ! lo0 proto tcp from any to any port 6000:6010
#FW1: hostname.if's
/etc/hostname.bge0
up
/etc/hostname.bge1
up
/etc/hostname.carp1
inet 10.0.0.47 255.255.255.0 10.0.0.255 vhid 2 advbase 1 advskew 1
carppeer 10.0.0.49 pass MYPASS
/etc/hostname.em1
inet 192.168.254.1 255.255.255.0 NONE
/etc/hostname.pfsync0
up syncdev em1 syncpeer 192.168.254.2
/etc/hostname.trunk0
trunkproto lacp trunkport bge0 trunkport bge1 10.0.0.48 netmask 255.255.255.0 up
#FW2: hostname.if's
/etc/hostname.bge0
up
/etc/hostname.bge1
up
/etc/hostname.carp1
inet 10.0.0.47 255.255.255.0 10.0.0.255 vhid 2 advbase 1 advskew 100
carppeer 10.0.0.48 pass MYPASS
/etc/hostname.em1
inet 192.168.254.2 255.255.255.0 NONE
/etc/hostname.pfsync0
up syncdev em1 syncpeer 192.168.254.1
/etc/hostname.trunk0
trunkproto lacp trunkport bge0 trunkport bge1 10.0.0.49 netmask 255.255.255.0 up
Both FW are Dell R200's with a few extra em interfaces
# dmesg
OpenBSD 5.0 (GENERIC.MP) #63: Wed Aug 17 10:14:30 MDT 2011
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4292550656 (4093MB)
avail mem = 4164161536 (3971MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcff9c000 (46 entries)
bios0: vendor Dell Inc. version "1.4.3" date 05/15/2009
bios0: Dell Inc. PowerEdge R200
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET MCFG WD__ SLIC ERST HEST BERT
EINJ SSDT SSDT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz, 3000.60 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu0: 6MB 64b/line 16-way L2 cache
cpu0: apic clock running at 333MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz, 3000.21 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,NXE,LONG
cpu1: 6MB 64b/line 16-way L2 cache
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
ioapic1: misconfigured as apic 0, remapped to apid 3
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus 2 (SBE0)
acpiprt3 at acpi0: bus 3 (PXHA)
acpiprt4 at acpi0: bus 4 (SBE4)
acpiprt5 at acpi0: bus 5 (SBE5)
acpiprt6 at acpi0: bus 6 (COMP)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 3000 MHz: speeds: 3000, 2667, 2333, 2000 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06:
msi, address 00:15:17:cc:bf:08
em1 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06:
msi, address 00:15:17:cc:bf:09
ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 6702PXH PCIE-PCIX" rev 0x09
pci3 at ppb2 bus 3
em2 at pci3 dev 2 function 0 "Intel PRO/1000MF (82545GM)" rev 0x04:
apic 3 int 3, address 00:04:23:d2:9b:73
ppb3 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02
pci4 at ppb3 bus 4
bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 16, address 00:25:64:3c:67:5f
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb4 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02
pci5 at ppb4 bus 5
bge1 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
(0x4201): apic 2 int 17, address 00:25:64:3c:67:60
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 21
uhci1 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 20
uhci2 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 21
ehci0 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 21
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci6 at ppb5 bus 6
vga1 at pci6 dev 5 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 19
drm0 at radeondrm0
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide0: using apic 2 int 23 for native-PCI interrupt
wd0 at pciide0 channel 0 drive 0: <WDC WD1602ABKS-18N8A0>
wd0: 16-sector PIO, LBA48, 152587MB, 312500000 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, DVD-ROM TS-L333A, D550> ATAPI
5/cdrom removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
wd1 at pciide0 channel 1 drive 0: <WDC WD1602ABKS-18N8A0>
wd1: 16-sector PIO, LBA48, 152587MB, 312500000 sectors
wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhub4 at uhub0 port 5 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
sd0 at scsibus2 targ 1 lun 0: <OPENBSD, SR RAID 1, 004> SCSI2 0/direct fixed
sd0: 30725MB, 512 bytes/sector, 62926077 sectors
root on wd0a (deb23c5900ffdb8c.a) swap on wd0b dump on wd0b
