>Synopsis: 6.8 panic _bus_dmamap_load_mbuf: no packet header
>Category: system
>Environment:
System : OpenBSD 6.8
Details : OpenBSD 6.8 (GENERIC.MP) #98: Sun Oct 4 18:13:26 MDT 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
Two Dell PE 2950 configured as CARP peers, panicked at approx. the same
time about one hour after upgrading from 6.7 to 6.8. Backtraces with
6.8 release kernel looked both like this (no keyboard interaction
possible,
trace typed from screenshot):
login: panic: _bus_dmamap_load_mbuf: no packet header
Stopped at db_enter+0x10: popq %rbx
TID PID UID PRFLAGS PFLAGS CPU COMMAND
60777 39157 77 0x100010 0 1 dhcpd
244419 53804 73 0x100010 0 3 syslogd
*519642 20976 0 0x14000 0x200 4 softnet
288645 79348 0 0x14000 0x200 5 systqmp
384262 47177 0 0x14000 0x200 2 systq
db_enter() at db_enter+0x10
_bus_dmamap_load_mbuf(ffffffff820e8ed8,ffff800000981e00,fffffd80c9a67d00,1) at
bus_dmamap_load_mbuf+0xfe
bnx_tx_encap(ffff800000066000,fffffd80c9a67d00,ffff80002654e05c) at
bnx_tx_encap+0xa7
bnx_start(ffff8000000662d8) at bnx_start+0x8f
ifq_serialize(ffff8000000662d8,ffff8000000663b8) at ifq_serialize+0xfd
taskq_thread(ffff800000030080) at taskq_thread+0x81
end_trace frame: 0x0, count: 8
What may be special about these systems is that dhcpd is running on top
of carp0. And perhaps the pf rules are a bit unusual (see below). Most
DHCP
requests are forwarded by remote switches via unicast, but occasionally
there
is a DHCP request from within the local broadcast domain of the carp0
interface, which seems to trigger the bug.
Looking through the commit logs the change in ip_carp.c 1.344 looks
like a good candidate. After a few test kernels the theory seems to
hold.
With serial console enabled and a test kernel built from CVS date
2020.05.21.22.00.00 using ip_carp.c 1.345 I got this backtrace:
ddb{2}> trace
refcnt_rele(deaf4152deaf41ba) at refcnt_rele+0x19 [machine/atomic.h:234]
pf_state_key_unref(deaf4152deaf4152) at pf_state_key_unref+0x21
[/usr/src/sys/net/pf.c:7437]
pf_pkt_addr_changed(fffffd800a79f200) at pf_pkt_addr_changed+0x2e
[/usr/src/sys/net/pf.c:7482]
if_enqueue(ffff800000066048,fffffd800a79f200) at if_enqueue+0x54
[/usr/src/sys/net/if.c:709]
carp_transmit(ffff800000d33800,ffff800000066048,fffffd800a79f200) at
carp_transmit+0xc7 [/usr/src/sys/netinet/ip_carp.c:2307]
carp_enqueue(ffff800000d33800,fffffd800a79f200) at carp_enqueue+0x6f
[/usr/src/sys/netinet/ip_carp.c:0]
ether_output(ffff800000d33800,fffffd800a78f800,ffff800026d94bd0,0) at
ether_output+0x7e [/usr/src/sys/net/if_ethersubr.c:343]
carp_output(ffff800000d33800,fffffd800a78f800,ffff800026d94bd0,0) at
carp_output+0x97 [/usr/src/sys/netinet/ip_carp.c:2358]
bpfwrite(21700,ffff800026d94f90,1) at bpfwrite+0x171
[/usr/src/sys/net/bpf.c:628]
spec_write(ffff800026d94db0) at spec_write+0x95
[/usr/src/sys/kern/spec_vnops.c:309]
VOP_WRITE(fffffd812cd6e5b8,ffff800026d94f90,1,fffffd812f7bd660) at
VOP_WRITE+0x4f [/usr/src/sys/kern/vfs_vops.c:268]
vn_write(fffffd81079c6628,ffff800026d94f90,0) at vn_write+0xcf
[/usr/src/sys/kern/vfs_vnops.c:414]
dofilewritev(ffff800026d05880,4,ffff800026d94f90,0,ffff800026d95090) at
dofilewritev+0x14d [/usr/src/sys/kern/sys_generic.c:366]
sys_writev(ffff800026d05880,ffff800026d95030,ffff800026d95090) at
sys_writev+0xe2 [/usr/src/sys/kern/sys_generic.c:312]
syscall(ffff800026d95100) at syscall+0x389
[/usr/src/sys/sys/syscall_mi.h:102]
Xsyscall() at Xsyscall+0x128
end of kernel
end trace frame: 0x7f7ffffdd990, count: -16
ddb{2}> ps
...
10143 46499 42996 95 3 0x100092 kqread smtpd
42996 392243 1 0 3 0x100080 kqread smtpd
*76283 434096 1 77 7 0x100010 dhcpd
21118 236803 1 0 3 0x80 select sshd
50078 303006 1 0 3 0x100080 poll ntpd
...
With a kernel checked out with CVS date 2020.05.21.03.00.00 using
ip_carp.c 1.343 the bug is not reproducible so far.
>How-To-Repeat:
dhcpd on carp0 and DHCP requests from within the broadcast domain of
the carp0 interface seems to be the trigger.
>Fix:
n/a
rc.conf.local:
dhcpd_flags="-y bnx1 -Y 10.1.7.12 carp0
pfctl -sr:
block drop in quick on egress inet proto tcp from 10.1.2.123 to any
port = 53
pass in quick on egress all no state
block return out log proto tcp all user = 55
block return out log proto udp all user = 55
dmesg:
OpenBSD 6.8 (GENERIC.MP) #98: Sun Oct 4 18:13:26 MDT 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4273274880 (4075MB)
avail mem = 4128714752 (3937MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcfb9c000 (66 entries)
bios0: vendor .... version "2.2.6" date 02/14/2008
bios0: ....
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET MCFG WD__ ERST HEST BERT EINJ TCPA
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.61 MHz, 06-17-06
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu0: 6MB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 332MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2, IBE
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.29 MHz, 06-17-06
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu1: 6MB 64b/line 16-way L2 cache
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.29 MHz, 06-17-06
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu2: 6MB 64b/line 16-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.29 MHz, 06-17-06
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu3: 6MB 64b/line 16-way L2 cache
cpu3: smt 0, core 2, package 1
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.26 MHz, 06-17-06
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu4: 6MB 64b/line 16-way L2 cache
cpu4: smt 0, core 1, package 0
cpu5 at mainbus0: apid 5 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.26 MHz, 06-17-06
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu5: 6MB 64b/line 16-way L2 cache
cpu5: smt 0, core 1, package 1
cpu6 at mainbus0: apid 3 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.26 MHz, 06-17-06
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu6: 6MB 64b/line 16-way L2 cache
cpu6: smt 0, core 3, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5440 @ 2.83GHz, 2826.26 MHz, 06-17-06
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR,MELTDOWN
cpu7: 6MB 64b/line 16-way L2 cache
cpu7: smt 0, core 3, package 1
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins, remapped
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (PEX2)
acpiprt2 at acpi0: bus 5 (UPST)
acpiprt3 at acpi0: bus 6 (DWN1)
acpiprt4 at acpi0: bus 8 (DWN2)
acpiprt5 at acpi0: bus 1 (PEX3)
acpiprt6 at acpi0: bus -1 (PE2P)
acpiprt7 at acpi0: bus 10 (PEX4)
acpiprt8 at acpi0: bus 12 (PEX6)
acpiprt9 at acpi0: bus 2 (SBEX)
acpiprt10 at acpi0: bus 14 (COMP)
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
acpicmos0 at acpi0
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x12
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x12
pci1 at ppb0 bus 4
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 5
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 6
ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci4 at ppb3 bus 7
bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int 16
ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01: msi
pci5 at ppb4 bus 8
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 9
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x12
pci7 at ppb6 bus 1
mfi0 at pci7 dev 0 function 0 "Symbios Logic SAS1078" rev 0x04: apic 8 int 16
mfi0: "PERC 6/i Integrated", firmware 6.0.3-0002, 256MB cache
scsibus1 at mfi0: 64 targets
sd0 at scsibus1 targ 0 lun 0: <DELL, PERC 6/i, 1.11>
naa.6001ec90ebc5aa0019814bce0764f745
sd0: 285568MB, 512 bytes/sector, 584843264 sectors
ppb7 at pci0 dev 4 function 0 "Intel 5000 PCIE x8" rev 0x12: msi
pci8 at ppb7 bus 10
em0 at pci8 dev 0 function 0 "Intel I350" rev 0x01: msi, address
a0:36:9f:07:43:c0
em1 at pci8 dev 0 function 1 "Intel I350" rev 0x01: msi, address
a0:36:9f:07:43:c1
em2 at pci8 dev 0 function 2 "Intel I350" rev 0x01: msi, address
a0:36:9f:07:43:c2
em3 at pci8 dev 0 function 3 "Intel I350" rev 0x01: msi, address
a0:36:9f:07:43:c3
ppb8 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x12
pci9 at ppb8 bus 11
ppb9 at pci0 dev 6 function 0 "Intel 5000 PCIE x8" rev 0x12: msi
pci10 at ppb9 bus 12
ppb10 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x12
pci11 at ppb10 bus 13
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x12
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x12
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x12
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x12
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x12
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x12
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x12
ppb11 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09
pci12 at ppb11 bus 2
ppb12 at pci12 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3
pci13 at ppb12 bus 3
bnx1 at pci13 dev 0 function 0 "Broadcom BCM5708" rev 0x12: apic 8 int 16
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 21
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 20
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 21
uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09: apic 8 int 20
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 21
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
ppb13 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
pci14 at ppb13 bus 14
radeondrm0 at pci14 dev 13 function 0 "ATI ES1000" rev 0x02
drm0 at radeondrm0
radeondrm0: apic 8 int 19
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, CD-ROM CD-224E-N, 3.AC> removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00
addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub5 at uhub0 port 5 configuration 1 interface 0 "Cypress Semiconductor USB2
Hub" rev 2.00/0.0b addr 2
uhidev0 at uhub2 port 1 configuration 1 interface 0 "Kb KVM -COMPOSITE" rev
1.10/1.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
uhidev1 at uhub2 port 1 configuration 1 interface 1 "Kb KVM -COMPOSITE" rev
1.10/1.00 addr 2
uhidev1: iclass 3/0, 2 report ids
uhid0 at uhidev1 reportid 1: input=2, output=0, feature=0
uhid1 at uhidev1 reportid 2: input=1, output=0, feature=0
uhidev2 at uhub2 port 1 configuration 1 interface 2 "Kb KVM -COMPOSITE" rev
1.10/1.00 addr 2
uhidev2: iclass 3/1
ums0 at uhidev2: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (11dd7e396d3fcc27.a) swap on sd0b dump on sd0b
bnx0: address 00:22:19:07:14:24
brgphy0 at bnx0 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
bnx1: address 00:22:19:07:14:22
brgphy1 at bnx1 phy 1: BCM5708C 10/100/1000baseT PHY, rev. 6
initializing kernel modesetting (RV100 0x1002:0x515E 0x1028:0x01B2 0x02).
[drm] *ERROR* radeon: ring test failed (scratch(0x15E4)=0xCAFEDEAD)
[drm] *ERROR* radeon: cp isn't working (-22).
drm:pid0:r100_startup *ERROR* failed initializing CP (-22).
drm:pid0:r100_init *ERROR* Disabling GPU acceleration
radeondrm0: 1024x768, 16bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wskbd1: connecting to wsdisplay0
wsdisplay0: screen 1-5 added (std, vt100 emulation)
