I have same errors in log and same behavior on one of my firewall
cluster. Carp flapping is a result of master system lagging and not
sending carp packets on time. You can workaround it by setting advbase
to some larger value, e.g. 10, but it will obviously still keep
lagging and dropping packets on NIC when these lags occur.
On my graphs I noticed peak in pf match ratio, peak in packesloss on
wan iface, higher cpu load and utilization when this ocucrs. I am not
graphing mbufs, it is around 500-900 usually, kern.maxclusters was set
to 32768 and and peak in netstat -m was 32770, so I raised that to
64000. It had no effect, it did not peak above 32770 and lags still
occur.
Dont really know what to do with this, I am sort of just waiting for
5.9 release and its unlocked networking stack hoping that multicore
cpu will manage that load.
On Thu, Jan 7, 2016 at 10:52 AM, Håkon Lerring <ha...@lerring.no> wrote:
>> On 05 Jan 2016, at 13:06, Stefan Sperling <s...@stsp.name> wrote:
>>
>> On Tue, Jan 05, 2016 at 12:29:43PM +0100, Håkon Lerring wrote:
>>> Hello misc.
>>>
>>> I was investigating a problem with a firewall that goes AWOL every week.
> It
>>> happens only if i activate an ipv6 address on a carp interface. The carp
> log
>>> has this message:
>>>
>>> Jan 5 12:10:06 <hostname> /bsd: carp: packet size 48 too small
>>>
>>> I think i have narrowed down the leak to the handling of too small
>>> ipv6-packets:
>>>
>>> --- ip_carp.c.orig 2016-01-05 12:18:03.000000000 +0100
>>> +++ ip_carp.c 2016-01-05 12:18:30.000000000 +0100
>>> @@ -562,6 +562,7 @@
>>> if ((m = m_pullup(m, *offp + sizeof(*ch))) == NULL) {
>>> carpstats.carps_badlen++;
>>> CARP_LOG(LOG_INFO, sc, ("packet size %u too small", len));
>>> + m_freem(m);
>>> return (IPPROTO_DONE);
>>> }
>>> ch = (struct carp_header *)(mtod(m, caddr_t) + *offp);
>>>
>>>
>>> I have not yet tested this patch since this is a production system. Why
> the
>>> other machine is sending incomplete packets is another question i'm
> currently
>>> investigating.
>>
>> Your patch effectively just calls m_freem(NULL);
>> And m_pullup already frees the mbuf on failure.
>>
>> Can you describe the actual problem you're seeing with more words than
> "AWOL"?
>
> Sorry for my hasty conclusion about my problem (i see that i might be wrong
> about the cause), i will explain it in more detail.
>
> I have 2 firewalls/routers with multiple carp interfaces (11). 1 on uplink
> (carp2) and the rest for different internal networks. All the carp
interfaces
> are set up as a preemptive failover in case the master goes down. The
problem
> has two symptoms but i believe they might have the same cause (as they
occur
> at the same time).
>
> The first symptom is that at what seems to be random times the carp2
interface
> (see ifconfig output below) on the backup-machine starts flapping between
> backup and master (the designated master does not do the same) and complain
> about too small packets:
>
> Jan 7 09:00:11 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan 7 09:00:12 hostname /bsd: carp2: state transition: MASTER -> BACKUP
> Jan 7 09:00:12 hostname /bsd: carp: packet size 48 too small
> Jan 7 09:00:18 hostname last message repeated 6 times
> Jan 7 09:00:19 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan 7 09:00:19 hostname /bsd: carp2: state transition: MASTER -> BACKUP
> Jan 7 09:00:19 hostname /bsd: carp: packet size 48 too small
> Jan 7 09:00:25 hostname last message repeated 6 times
> Jan 7 09:00:26 hostname /bsd: carp2: state transition: BACKUP -> MASTER
> Jan 7 09:00:26 hostname /bsd: carp2: state transition: MASTER -> BACKUP
>
> etc..
>
> The designated master does also log that "packet size 48 too small"
>
> MASTER carp2:
> carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 00:00:5e:00:01:02
> description: uplink
> priority: 15
> carp: MASTER carpdev bnx0 vhid 2 advbase 1 advskew 0
> groups: carp
> status: master
> inet <shared IP> netmask 0xfffffff8 broadcast <broadcast>
> inet6 fe80::200:5eff:fe00:102%carp2 prefixlen 64 scopeid 0x16
> inet6 <shared ipv6> prefixlen 64
>
> BACKUP carp2:
> carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 00:00:5e:00:01:02
> description: uplink
> priority: 15
> carp: BACKUP carpdev bnx0 vhid 2 advbase 2 advskew 128
> groups: carp
> status: backup
> inet <shared IP> netmask 0xfffffff8 broadcast <broadcast>
> inet6 fe80::200:5eff:fe00:102%carp2 prefixlen 64 scopeid 0x16
> inet6 <shared ipv6> prefixlen 64
>
> The other carp interfaces on the same machine does not have this problem.
The
> only things different about carp2 from the other interfaces is that carp2
has
> an IPv6 address and is backed by a bnx interface instead of an em interface
> (though this does not seem to be of significance as i have tried it with em
> interface with the same result).
>
> The other symptom (which causes most problems) is that at the same time the
> amount mbuf clusters starts to increase. I set kern.maxclusters=12288 to
stop
> it from running out too quickly but i don't think it will ever be satisfied
if
> i continue to increase the limit. When it runs out of mbuf clusters it just
> stops forwarding packets and SSH drops any connection attempts. Here is a
> munin graph from this morning:
> https://www.dropbox.com/s/j4353z4nl9e22jg/netmem_clusters.png?dl=0
> <https://www.dropbox.com/s/j4353z4nl9e22jg/netmem_clusters.png?dl=0> (
sorry
> archive users for linking on a mailing list )
>
> The symptoms sometimes disappear after reboot but ocurr again after a
while.
>
> Environment:
> System : OpenBSD 5.8
> Details : OpenBSD 5.8 (GENERIC.MP) #1236: Sun Aug 16 02:31:04
MDT
> 2015
>
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/co
> mpile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine : amd64
>
> dmesg:
> OpenBSD 5.8 (GENERIC.MP) #1236: Sun Aug 16 02:31:04 MDT 2015
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4276432896 (4078MB)
> avail mem = 4142936064 (3951MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xee000 (68 entries)
> bios0: vendor HP version "P58" date 11/02/2008
> bios0: HP ProLiant DL360 G5
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPCR MCFG HPET SPMI ERST APIC FFFF BERT HEST SSDT
> acpi0: wakeup devices PCI0(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
> acpihpet0 at acpi0: 14318179 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2833.77 MHz
> cpu0:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM
> 2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
> cpu0: 6MB 64b/line 16-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 333MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.2.2, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2000.08 MHz
> cpu1:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM
> 2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
> cpu1: 6MB 64b/line 16-way L2 cache
> cpu1: smt 0, core 2, package 0
> cpu2 at mainbus0: apid 1 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2833.44 MHz
> cpu2:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM
> 2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
> cpu2: 6MB 64b/line 16-way L2 cache
> cpu2: smt 0, core 1, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2656.37 MHz
> cpu3:
>
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
>
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM
> 2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF,SENSOR
> cpu3: 6MB 64b/line 16-way L2 cache
> cpu3: smt 0, core 3, package 0
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
> acpiprt0 at acpi0: bus 1 (IP2P)
> acpiprt1 at acpi0: bus 11 (IPE1)
> acpiprt2 at acpi0: bus 10 (IPE4)
> acpiprt3 at acpi0: bus 16 (P2P2)
> acpiprt4 at acpi0: bus 9 (PT02)
> acpiprt5 at acpi0: bus 6 (PT03)
> acpiprt6 at acpi0: bus 19 (PT04)
> acpiprt7 at acpi0: bus 3 (NB01)
> acpiprt8 at acpi0: bus 5 (NB02)
> acpiprt9 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: !C3(250@17 io@0x915), !C2(500@1 io@0x914), C1(1000@1
halt)
> acpicpu1 at acpi0: !C3(250@17 io@0x915), !C2(500@1 io@0x914), C1(1000@1
halt)
> acpicpu2 at acpi0: !C3(250@17 io@0x915), !C2(500@1 io@0x914), C1(1000@1
halt)
> acpicpu3 at acpi0: !C3(250@17 io@0x915), !C2(500@1 io@0x914), C1(1000@1
halt)
> acpitz0 at acpi0: critical temperature is 31 degC
> ipmi at mainbus0 not configured
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 5000P Host" rev 0xb1
> ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0xb1
> pci1 at ppb0 bus 9
> ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci2 at ppb1 bus 10
> ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci3 at ppb2 bus 11
> em0 at pci3 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 8 int 16,
address
> 00:1f:29:61:57:4a
> em1 at pci3 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 8 int 17,
address
> 00:1f:29:61:57:4b
> ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci4 at ppb3 bus 14
> ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci5 at ppb4 bus 15
> ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
> pci6 at ppb5 bus 16
> ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1
> pci7 at ppb6 bus 6
> ppb7 at pci7 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb4
> pci8 at ppb7 bus 7
> ppb8 at pci8 dev 4 function 0 "ServerWorks HT-1000 PCIX" rev 0xb2
> pci9 at ppb8 bus 8
> ciss0 at pci8 dev 8 function 0 "Hewlett-Packard Smart Array" rev 0x00: apic
8
> int 16
> ciss0: 1 LD, HW rev 0, FW 1.80/1.80
> scsibus1 at ciss0: 1 targets
> sd0 at scsibus1 targ 0 lun 0: <HP, LOGICAL VOLUME, 1.80> SCSI3 0/direct
fixed
> sd0: 69973MB, 512 bytes/sector, 143305920 sectors
> ppb9 at pci0 dev 4 function 0 "Intel 5000 PCIE x8" rev 0xb1
> pci10 at ppb9 bus 19
> ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0xb1
> pci11 at ppb10 bus 22
> ppb11 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0xb1
> pci12 at ppb11 bus 2
> ppb12 at pci12 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci13 at ppb12 bus 3
> bnx0 at pci13 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int 18
> ppb13 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0xb1
> pci14 at ppb13 bus 4
> ppb14 at pci14 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
> pci15 at ppb14 bus 5
> bnx1 at pci15 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int 19
> pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0xb1
> pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0xb1
> pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0xb1
> pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1
> pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1
> pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1
> pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1
> uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 16
> uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 17
> uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 18
> uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: apic 8 int 19
> ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 16
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb15 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
> pci16 at ppb15 bus 1
> radeondrm0 at pci16 dev 3 function 0 "ATI ES1000" rev 0x02
> drm0 at radeondrm0
> radeondrm0: apic 8 int 23
> "Compaq iLO" rev 0x03 at pci16 dev 4 function 0 not configured
> "Compaq iLO" rev 0x03 at pci16 dev 4 function 2 not configured
> uhci4 at pci16 dev 4 function 4 "Hewlett-Packard USB" rev 0x00: apic 8 int
22
> "Hewlett-Packard IPMI" rev 0x00 at pci16 dev 4 function 6 not configured
> usb1 at uhci4: USB revision 1.0
> uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1
> pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
> pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel
0
> configured to compatibility, channel 1 configured to compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus2 at atapiscsi0: 2 targets
> cd0 at scsibus2 targ 0 lun 0: <HL-DT-ST, DVDRAM GSA-T40L, KS03> ATAPI
5/cdrom
> removable
> cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> pciide0: channel 1 disabled (no drives)
> usb2 at uhci0: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci1: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb4 at uhci2: USB revision 1.0
> uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb5 at uhci3: USB revision 1.0
> uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> com1: probed fifo depth: 0 bytes
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual Keyboard"
rev
> 1.10/0.02 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
> wskbd1 at ukbd0 mux 1
> uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual Keyboard"
rev
> 1.10/0.02 addr 2
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons
> wsmouse0 at ums0 mux 0
> uhub6 at uhub1 port 2 "HP Virtual Hub" rev 1.10/0.01 addr 3
> uhidev2 at uhub3 port 2 configuration 1 interface 0 "Avocent Dell 03R874"
rev
> 1.10/1.00 addr 2
> uhidev2: iclass 3/1
> ukbd1 at uhidev2: 8 variable keys, 6 key codes, country code 33
> wskbd2 at ukbd1 mux 1
> uhidev3 at uhub3 port 2 configuration 1 interface 1 "Avocent Dell 03R874"
rev
> 1.10/1.00 addr 2
> uhidev3: iclass 3/1, 3 report ids
> ums1 at uhidev3 reportid 1: 5 buttons, Z dir
> wsmouse1 at ums1 mux 0
> uhid0 at uhidev3 reportid 2: input=2, output=0, feature=0
> uhid1 at uhidev3 reportid 3: input=1, output=0, feature=0
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (0f18f0dd8d57a1aa.a) swap on sd0b dump on sd0b
> bnx0: address 00:23:7d:33:11:c2
> brgphy0 at bnx0 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
> bnx1: address 00:23:7d:33:11:c0
> brgphy1 at bnx1 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
> error: [drm:pid0:r100_cp_init_microcode] *ERROR* radeon_cp: Failed to load
> firmware "radeon-r100_cp"
> error: [drm:pid0:r100_cp_init] *ERROR* Failed to load firmware!
> drm:pid0:r100_startup *ERROR* failed initializing CP (-2).
> drm:pid0:r100_init *ERROR* Disabling GPU acceleration
> radeondrm0: 1024x768
> wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using
wskbd0
> wskbd1: connecting to wsdisplay0
> wskbd2: connecting to wsdisplay0
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> carp: packet size 48 too small
> carp: packet size 48 too small
> <REMOVED 1030 REPEATED LINES>
> carp: packet size 48 too small
> carp: packet size 48 too small
> carp: packet size 48 too small
