Hi,
One of my machines experienced the following this morning:
kernel: protection fault trap, code=0
Stopped at pf_counters_inc+0x17c: movq 0x8(%rbx),%rax
This is unfortunately way beyond my level to even attempt to fix,
but after discussing this with Renaud Allard, his notes were:
Looking at the code at pf.c:8177-8180:
SLIST_FOREACH(sni, &st->src_nodes, next) {
sni->sn->packets[dirndx]++; // <- LINE 8178
sni->sn->bytes[dirndx] += pd->tot_len;
}
The crash is happening when:
1. %rbx contains the sni pointer (the list item)
2. movq 0x8(%rbx),%rax is dereferencing sni->sn
(the source node pointer at offset +8)
3. But sni has already been freed by another thread in pf_lb.c:301
Timeline:
Time Thread A (packet path) Thread B (cleanup)
---- ------------------------------ --------------------------------
T1 pf_counters_inc() called
T2 SLIST_FOREACH(sni, ...)
T3 -> sni points to valid item
T4 pf_map_addr_sticky() called
T5 -> pf_state_rm_src_node(s, sn)
T6 -> SLIST_REMOVE(sni from list)
T7 -> pool_put(&pf_sn_item_pl, sni)
T8 -> sni memory is FREED
T9 movq 0x8(%rbx),%rax <- CRASH!
(trying to read sni->sn)
This is a virtual machine at Vultr running with hw.smt=1, not sure if
that makes any difference, but might worth mentioning. dmesg(8) below:
Jesper Wallin
OpenBSD 7.8 (GENERIC.MP) #54: Sun Oct 12 12:58:11 MDT 2025
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278042624 (4079MB)
avail mem = 4121710592 (3930MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC-Rome Processor, 1996.87 MHz, 17-31-00
cpu0: cpuid 1
edx=178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT>
ecx=f6f83203<SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV>
cpu0: cpuid 6 eax=4<ARAT>
cpu0: cpuid 7.0 ebx=218001a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,CLFLUSHOPT,CLWB,SHA>
ecx=400004<UMIP> edx=84000000<IBRS,IBPB,SSBD>
cpu0: cpuid d.1 eax=9<XSAVEOPT,XSAVES>
cpu0: cpuid 80000001 edx=2fd3fbff<NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG>
ecx=c002f3<LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,OSVW,TOPEXT,CPCTR>
cpu0: cpuid 80000008 ebx=9205<IBPB,STIBP>
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line
8-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD EPYC-Rome Processor, 1997.27 MHz, 17-31-00
cpu1: smt 1, core 0, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
acpicmos0 at acpi0
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 2 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci1 at ppb0 bus 1
virtio0 at pci1 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio0 at virtio0: 1 queue, address 56:00:05:87:38:9f
virtio0: msix
ppb1 at pci0 dev 2 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Red Hat PCI" rev 0x00
pci3 at ppb2 bus 3
"Intel 6300ESB WDT" rev 0x00 at pci3 dev 1 function 0 not configured
ppb3 at pci0 dev 2 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci4 at ppb3 bus 4
xhci0 at pci4 dev 0 function 0 "Red Hat xHCI" rev 0x01: msix, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Red Hat xHCI root hub" rev 3.00/1.00
addr 1
ppb4 at pci0 dev 2 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci5 at ppb4 bus 5
virtio1 at pci5 dev 0 function 0 "Qumranet Virtio 1.x Storage" rev 0x01
vioblk0 at virtio1
virtio1: msix per-VQ
scsibus1 at vioblk0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 102400MB, 512 bytes/sector, 209715200 sectors
ppb5 at pci0 dev 2 function 4 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci6 at ppb5 bus 6
virtio2 at pci6 dev 0 function 0 "Qumranet Virtio 1.x Memory Balloon" rev 0x01
viomb0 at virtio2
virtio2: apic 0 int 22
ppb6 at pci0 dev 2 function 5 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci7 at ppb6 bus 7
virtio3 at pci7 dev 0 function 0 "Qumranet Virtio 1.x RNG" rev 0x01
viornd0 at virtio3
virtio3: msix per-VQ
ppb7 at pci0 dev 2 function 6 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci8 at ppb7 bus 8
ppb8 at pci0 dev 2 function 7 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci9 at ppb8 bus 9
ppb9 at pci0 dev 3 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci10 at ppb9 bus 10
ppb10 at pci0 dev 3 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci11 at ppb10 bus 11
ppb11 at pci0 dev 3 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci12 at ppb11 bus 12
ppb12 at pci0 dev 3 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci13 at ppb12 bus 13
ppb13 at pci0 dev 3 function 4 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci14 at ppb13 bus 14
ppb14 at pci0 dev 3 function 5 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci15 at ppb14 bus 15
ppb15 at pci0 dev 3 function 6 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci16 at ppb15 bus 16
ppb16 at pci0 dev 3 function 7 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci17 at ppb16 bus 17
ppb17 at pci0 dev 4 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci18 at ppb17 bus 18
ppb18 at pci0 dev 4 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci19 at ppb18 bus 19
ppb19 at pci0 dev 4 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci20 at ppb19 bus 20
ppb20 at pci0 dev 4 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci21 at ppb20 bus 21
ppb21 at pci0 dev 4 function 4 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci22 at ppb21 bus 22
ppb22 at pci0 dev 4 function 5 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci23 at ppb22 bus 23
ppb23 at pci0 dev 4 function 6 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci24 at ppb23 bus 24
ppb24 at pci0 dev 4 function 7 "Red Hat PCIE" rev 0x00: apic 0 int 20
pci25 at ppb24 bus 25
ppb25 at pci0 dev 5 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci26 at ppb25 bus 26
ppb26 at pci0 dev 5 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci27 at ppb26 bus 27
ppb27 at pci0 dev 5 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci28 at ppb27 bus 28
ppb28 at pci0 dev 5 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci29 at ppb28 bus 29
ppb29 at pci0 dev 5 function 4 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci30 at ppb29 bus 30
ppb30 at pci0 dev 5 function 5 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci31 at ppb30 bus 31
ppb31 at pci0 dev 5 function 6 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci32 at ppb31 bus 32
ppb32 at pci0 dev 5 function 7 "Red Hat PCIE" rev 0x00: apic 0 int 21
pci33 at ppb32 bus 33
ppb33 at pci0 dev 6 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci34 at ppb33 bus 34
azalia0 at pci0 dev 27 function 0 "Intel 82801I HD Audio" rev 0x03: msi
azalia0: No codecs found
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
ahci0: port 2: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 2 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub0 port 5 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev
2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (f8661402150bce7b.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
