My router has become unstable since upgrading from 7.1-stable to 7.2. After several days of uptime the machine gets into a state where some applications (unbound & dhcpd) report ENOBUFS (No buffer space available). At that time the machine is pingable over all the interfaces, but only the upstream interface seems functional (igc0). The networks downstream of the router can't get much data across. I don't have a good characterization of this.
At first I suspected this had something to do with the igc checksum offloading commit, so I am now running 7.2 with this reverted: "Implement and enable IPv4, TCP, and UDP checksum offloading for igc." I also started monitoring some counters that appeared relevant with this trivial loop: $ while : ; do date; netstat -s | grep err; netstat -m; netstat -ni | grep '^[Ni]'; sleep 300; done | tee err-log I have some 38 hours worth of counters as of now. I observe an upward trend in "mbuf 2112" and "mbufs in use", I extracted the values with $ perl -ne 'print "$x,$1\n" if m/^(\d+).*mbuf 2112/; $x=$1 if /^(\d+)\smbufs in use/;' err-log It starts out 610,410-ish and ends at 717,513. I have a picture for those visually inclined: https://photos.app.goo.gl/DZGCrJnJDohPrVyZ8 I have dmesg at the end and I shared the hostname files with mbuhl@. There's nothing special there AFAICT. The pf ruleset is also a pretty trivial NAT setup with DNS rdr-to to local unbound. I'll keep monitoring these numbers and am happy to share the full file with people who care to look at raw data. If there's more data to collect, please suggest what it is. If the symptoms happen again I may have to go back to 7.1 to avoid my family revolting. This will give me a baseline of the same measurements if nothing else. Thanks Greg OpenBSD 7.2 (GENERIC.MP) #0: Fri Oct 28 19:14:33 PDT 2022 greg@local:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4093276160 (3903MB) avail mem = 3951816704 (3768MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 3.2 @ 0xeae90 (57 entries) bios0: vendor American Megatrends Inc. version "5.13" date 01/06/2022 bios0: Default string Default string acpi0 at bios0: ACPI 6.2 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP FPDT FIDT MCFG SSDT DBG2 DBGP HPET LPIT APIC NPKT SSDT SSDT SSDT SSDT SSDT SSDT SSDT TPM2 DMAR WDAT WSMT acpi0: wakeup devices SIO1(S3) HDAS(S3) XHC_(S4) XDCI(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimcfg0 at acpi0 acpimcfg0: addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 19200000 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz, 1994.48 MHz, 06-7a-08 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 19MHz cpu0: mwait min=64, max=64, C-substates=0.2.0.2.4.2.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz, 1994.48 MHz, 06-7a-08 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz, 1994.48 MHz, 06-7a-08 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Celeron(R) J4125 CPU @ 2.00GHz, 1994.48 MHz, 06-7a-08 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache cpu3: smt 0, core 3, package 0 ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 120 pins acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 5 (RP01) acpiprt2 at acpi0: bus 6 (RP02) acpiprt3 at acpi0: bus 1 (RP03) acpiprt4 at acpi0: bus 2 (RP04) acpiprt5 at acpi0: bus 3 (RP05) acpiprt6 at acpi0: bus 4 (RP06) acpiec0 at acpi0: not present acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000 com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo com0: probed fifo depth: 15 bytes acpicmos0 at acpi0 acpibtn0 at acpi0: PWRB "INT33A1" at acpi0 not configured tpm0 at acpi0 TPM_ 2.0 (CRB) addr 0xfed40000/0x5000, device 0x00000000 rev 0x0 "PNP0C0B" at acpi0 not configured acpipwrres0 at acpi0: DRST acpipwrres1 at acpi0: DRST acpipwrres2 at acpi0: DRST acpipwrres3 at acpi0: DRST acpipwrres4 at acpi0: DRST acpipwrres5 at acpi0: DRST acpipwrres6 at acpi0: WRST acpicpu0 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS acpicpu1 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS acpicpu2 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS acpicpu3 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS acpipwrres7 at acpi0: FN00, resource for FAN0 acpitz0 at acpi0: critical temperature is 95 degC acpivideo0 at acpi0: GFX0 acpivout0 at acpivideo0: DD1F cpu0: Enhanced SpeedStep 1994 MHz: speeds: 2001, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel Gemini Lake Host" rev 0x06 inteldrm0 at pci0 dev 2 function 0 "Intel UHD Graphics 600" rev 0x06 drm0 at inteldrm0 inteldrm0: msi, GEMINILAKE, gen 9 azalia0 at pci0 dev 14 function 0 "Intel Gemini Lake HD Audio" rev 0x06: msi azalia0: no supported codecs "Intel Gemini Lake MEI" rev 0x06 at pci0 dev 15 function 0 not configured vendor "Intel", unknown product 0x31a2 (class prehistoric unknown subclass 0x50, rev 0x06) at pci0 dev 17 function 0 not configured ahci0 at pci0 dev 18 function 0 "Intel Gemini Lake AHCI" rev 0x06: msi, AHCI 1.3.1 ahci0: PHY offline on port 0 ahci0: port 1: 6.0Gb/s scsibus1 at ahci0: 32 targets sd0 at scsibus1 targ 1 lun 0: <ATA, CWDISK 64GB, U080> t10.ATA_CWDISK_64GB_AA000000000000001168 sd0: 61057MB, 512 bytes/sector, 125045424 sectors, thin ppb0 at pci0 dev 19 function 0 "Intel Gemini Lake PCIE" rev 0xf6: msi pci1 at ppb0 bus 1 igc0 at pci1 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4 queues, address 00:e2:69: ppb1 at pci0 dev 19 function 1 "Intel Gemini Lake PCIE" rev 0xf6: msi pci2 at ppb1 bus 2 igc1 at pci2 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4 queues, address 00:e2:69: ppb2 at pci0 dev 19 function 2 "Intel Gemini Lake PCIE" rev 0xf6: msi pci3 at ppb2 bus 3 igc2 at pci3 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4 queues, address 00:e2:69: ppb3 at pci0 dev 19 function 3 "Intel Gemini Lake PCIE" rev 0xf6: msi pci4 at ppb3 bus 4 igc3 at pci4 dev 0 function 0 "Intel I225-V" rev 0x03, msix, 4 queues, address 00:e2:69: ppb4 at pci0 dev 20 function 0 "Intel Gemini Lake PCIE" rev 0xf6 pci5 at ppb4 bus 5 ppb5 at pci0 dev 20 function 1 "Intel Gemini Lake PCIE" rev 0xf6 pci6 at ppb5 bus 6 xhci0 at pci0 dev 21 function 0 "Intel Gemini Lake xHCI" rev 0x06: msi, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1 pcib0 at pci0 dev 31 function 0 "Intel Gemini Lake LPC" rev 0x06 ichiic0 at pci0 dev 31 function 1 "Intel Gemini Lake SMBus" rev 0x06: apic 1 int 20 iic0 at ichiic0 spdmem0 at iic0 addr 0x50: 4GB DDR4 SDRAM PC4-19200 SO-DIMM isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard pcppi0 at isa0 port 0x61 spkr0 at pcppi0 vmm0 at mainbus0: VMX/EPT efifb at mainbus0 not configured
