I've manage to reproduce TSO em problem on anoter setup, unfortunatly
production.
Setup is very simple
em0 - carp <- uplink
em1 - pfsync
ix1 - vlans - carp
Jan 26 11:19:23 bcbnfw1 /bsd: em0: watchdog: head 34 tail 98 TDH 98 TDT 34
Jan 26 11:19:33 bcbnfw1 /bsd: em0: watchdog: head 345 tail 409 TDH 409
TDT 345
Jan 26 11:19:54 bcbnfw1 /bsd: em0: watchdog: head 259 tail 323 TDH 323
TDT 259
Jan 26 11:20:08 bcbnfw1 /bsd: em0: watchdog: head 343 tail 407 TDH 407
TDT 343
Jan 26 11:20:24 bcbnfw1 /bsd: em0: watchdog: head 20 tail 85 TDH 85 TDT 20
Jan 26 11:20:47 bcbnfw1 /bsd: em0: watchdog: head 388 tail 452 TDH 452
TDT 388
Jan 26 11:21:09 bcbnfw1 /bsd: em0: watchdog: head 25 tail 89 TDH 89 TDT 25
Jan 26 11:21:32 bcbnfw1 /bsd: em0: watchdog: head 105 tail 169 TDH 169
TDT 105
Jan 26 11:21:52 bcbnfw1 /bsd: em0: watchdog: head 23 tail 88 TDH 88 TDT 23
647470: Jan 26 11:19:25: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647474: Jan 26 11:19:29: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647478: Jan 26 11:19:35: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647483: Jan 26 11:19:39: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647487: Jan 26 11:19:56: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647491: Jan 26 11:19:59: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647495: Jan 26 11:20:10: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647499: Jan 26 11:20:13: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647504: Jan 26 11:20:26: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647508: Jan 26 11:20:29: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647512: Jan 26 11:20:49: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647516: Jan 26 11:20:52: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647520: Jan 26 11:21:11: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647524: Jan 26 11:21:14: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647528: Jan 26 11:21:34: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647532: Jan 26 11:21:36: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
647536: Jan 26 11:21:54: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to down
647540: Jan 26 11:21:56: %LINK-SP-3-UPDOWN: Interface
GigabitEthernet4/48, changed state to up
bcbnfw1# kstat em0:::
em0:0:em-stats:0
rx crc errs: 0 packets
rx align errs: 0 packets
rx align errs: 0 packets
rx errs: 0 packets
rx missed: 0 packets
tx single coll: 0 packets
tx excess coll: 0 packets
tx multi coll: 0 packets
tx late coll: 0 packets
tx coll: 0
tx defers: 0
tx no CRS: 0 packets
seq errs: 0
carr ext errs: 0 packets
rx len errs: 0 packets
rx xon: 0 packets
tx xon: 0 packets
rx xoff: 0 packets
tx xoff: 0 packets
FC unsupported: 0 packets
rx 64B: 6555 packets
rx 65-127B: 11144 packets
rx 128-255B: 6264 packets
rx 256-511B: 2390 packets
rx 512-1023B: 3706 packets
rx 1024-maxB: 87987 packets
rx good: 118046 packets
rx bcast: 3 packets
rx mcast: 82 packets
tx good: 56796 packets
rx good: 132532686 bytes
tx good: 13691390 bytes
rx no buffers: 0 packets
rx undersize: 0 packets
rx fragments: 0 packets
rx oversize: 0 packets
rx jabbers: 0 packets
rx mgmt: 0 packets
rx mgmt drops: 0 packets
tx mgmt: 0 packets
rx total: 132532686 bytes
tx total: 13691390 bytes
rx total: 118046 packets
tx total: 56796 packets
tx 64B: 11861 packets
tx 65-127B: 28718 packets
tx 128-255B: 7202 packets
tx 256-511B: 1834 packets
tx 512-1023B: 2059 packets
tx 1024-maxB: 5122 packets
tx mcast: 18 packets
tx bcast: 2 packets
em0:0:rxq:0
packets: 1009629 packets
bytes: 1172569417 bytes
fdrops: 0 packets
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
enqueues: 348100
dequeues: 348031
em0:0:txq:0
packets: 465709 packets
bytes: 103430590 bytes
qdrops: 53674 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 511 packets
oactive: false
oactives: 9
OpenBSD 7.4-current (GENERIC.MP) #1626: Thu Jan 25 20:05:01 MST 2024
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 34224844800 (32639MB)
avail mem = 33166430208 (31629MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xec9b0 (62 entries)
bios0: vendor American Megatrends Inc. version "3.4" date 06/05/2021
bios0: Supermicro X10SRW-F
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI HPET NFIT WDDT
SSDT NITR SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
acpi0: wakeup devices IP2P(S4) EHC1(S4) EHC2(S4) RP01(S4) RP02(S4)
RP03(S4) RP04(S4) RP05(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4)
BR2A(S4) BR2B(S4) BR2C(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.09 MHz, 06-4f-01,
patch 0b000040
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.18 MHz, 06-4f-01,
patch 0b000040
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.22 MHz, 06-4f-01,
patch 0b000040
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.34 MHz, 06-4f-01,
patch 0b000040
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.42 MHz, 06-4f-01,
patch 0b000040
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, 3600.46 MHz, 06-4f-01,
patch 0b000040
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu5: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache
cpu5: smt 0, core 5, package 0
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 2 pa 0xfec01000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0x80000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 255 (UNC0)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 4 (RP03)
acpiprt3 at acpi0: bus 5 (BR7A)
acpiprt4 at acpi0: bus 6 (RP05)
acpiprt5 at acpi0: bus 1 (BR1A)
acpiprt6 at acpi0: bus -1 (BR1B)
acpiprt7 at acpi0: bus -1 (BR2A)
acpiprt8 at acpi0: bus -1 (BR2C)
acpiprt9 at acpi0: bus 2 (BR3A)
acpiprt10 at acpi0: bus -1 (BR3C)
acpipci0 at acpi0 UNC0: 0x00000010 0x00000011 0x00000000
"ACPI0004" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
acpicmos0 at acpi0
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
com1: console
"IPI0001" at acpi0 not configured
acpibtn0 at acpi0: PWRB
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"ACPI000D" at acpi0 not configured
acpicpu0 at acpi0: C1(1016@1 mwait.1), PSS
acpicpu1 at acpi0: C1(1016@1 mwait.1), PSS
acpicpu2 at acpi0: C1(1016@1 mwait.1), PSS
acpicpu3 at acpi0: C1(1016@1 mwait.1), PSS
acpicpu4 at acpi0: C1(1016@1 mwait.1), PSS
acpicpu5 at acpi0: C1(1016@1 mwait.1), PSS
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 3600 MHz: speeds: 3601, 3600, 3400, 3300, 3100,
2900, 2700, 2600, 2400, 2200, 2100, 1900, 1700, 1500, 1400, 1200 MHz
pci0 at mainbus0 bus 255
255:30:3: mem address conflict 0x20/0x20
"Intel Xeon-D QPI Link" rev 0x01 at pci0 dev 11 function 0 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci0 dev 11 function 1 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci0 dev 11 function 2 not configured
"Intel Xeon-D QPI Debug" rev 0x01 at pci0 dev 11 function 3 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 12 function 0 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 12 function 1 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 12 function 2 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 12 function 3 not configured
"Intel E5 v4 Cache" rev 0x01 at pci0 dev 12 function 4 not configured
"Intel E5 v4 Cache" rev 0x01 at pci0 dev 12 function 5 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 15 function 0 not configured
"Intel E5 v4 Cache" rev 0x01 at pci0 dev 15 function 1 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 15 function 4 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 15 function 5 not configured
"Intel Xeon-D Cache" rev 0x01 at pci0 dev 15 function 6 not configured
"Intel Xeon-D PCIE" rev 0x01 at pci0 dev 16 function 0 not configured
"Intel E5 v4 R2PCIe Agent" rev 0x01 at pci0 dev 16 function 1 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci0 dev 16 function 5 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci0 dev 16 function 6 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci0 dev 16 function 7 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci0 dev 18 function 0 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci0 dev 18 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 4 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 5 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 19 function 7 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 4 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 5 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 20 function 7 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 21 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 21 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 21 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci0 dev 21 function 3 not configured
"Intel E5 v4 RAS" rev 0x01 at pci0 dev 22 function 0 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 22 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 22 function 7 not configured
"Intel E5 v4 Thermal" rev 0x01 at pci0 dev 23 function 0 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 23 function 4 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 23 function 5 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 23 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci0 dev 23 function 7 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 30 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 30 function 1 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 30 function 2 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 30 function 3 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 30 function 4 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 31 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci0 dev 31 function 2 not configured
pci1 at mainbus0 bus 0
pchb0 at pci1 dev 0 function 0 "Intel Xeon-D Host" rev 0x01
ppb0 at pci1 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x01: msi
pci2 at ppb0 bus 1
ix0 at pci2 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues,
address 90:e2:ba:d7:1b:f4
ix1 at pci2 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues,
address 90:e2:ba:d7:1b:f5
ppb1 at pci1 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x01: msi
pci3 at ppb1 bus 2
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 0 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 1 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 2 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 3 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 4 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 5 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 6 not configured
"Intel E5 v4 DMA" rev 0x01 at pci1 dev 4 function 7 not configured
"Intel Xeon-D Address Map" rev 0x01 at pci1 dev 5 function 0 not configured
"Intel Xeon-D Hot Plug" rev 0x01 at pci1 dev 5 function 1 not configured
"Intel Xeon-D RAS" rev 0x01 at pci1 dev 5 function 2 not configured
"Intel Xeon-D I/O APIC" rev 0x01 at pci1 dev 5 function 4 not configured
"Intel C610 MS SPSR" rev 0x05 at pci1 dev 17 function 0 not configured
ahci0 at pci1 dev 17 function 4 "Intel C610 AHCI" rev 0x05: msi, AHCI 1.3
scsibus1 at ahci0: 32 targets
xhci0 at pci1 dev 20 function 0 "Intel C610 xHCI" rev 0x05: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
"Intel C610 MEI" rev 0x05 at pci1 dev 22 function 0 not configured
"Intel C610 MEI" rev 0x05 at pci1 dev 22 function 1 not configured
ehci0 at pci1 dev 26 function 0 "Intel C610 USB" rev 0x05: apic 1 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb2 at pci1 dev 28 function 0 "Intel C610 PCIE" rev 0xd5
pci4 at ppb2 bus 3
ppb3 at pci1 dev 28 function 2 "Intel C610 PCIE" rev 0xd5: msi
pci5 at ppb3 bus 4
ppb4 at pci5 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci6 at ppb4 bus 5
vga1 at pci6 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay at vga1 not configured
ppb5 at pci1 dev 28 function 4 "Intel C610 PCIE" rev 0xd5: msi
pci7 at ppb5 bus 6
em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
0c:c4:7a:da:cd:5a
em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
0c:c4:7a:da:cd:5b
ehci1 at pci1 dev 29 function 0 "Intel C610 USB" rev 0x05: apic 1 int 18
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
pcib0 at pci1 dev 31 function 0 "Intel C610 LPC" rev 0x05
ahci1 at pci1 dev 31 function 2 "Intel C610 AHCI" rev 0x05: msi, AHCI 1.3
ahci1: port 0: 6.0Gb/s
ahci1: port 1: 6.0Gb/s
scsibus2 at ahci1: 32 targets
sd0 at scsibus2 targ 0 lun 0: <ATA, SAMSUNG MZ7LN256, MAV0>
naa.5002538d417c7a2b
sd0: 244198MB, 512 bytes/sector, 500118192 sectors, thin
sd1 at scsibus2 targ 1 lun 0: <ATA, SAMSUNG MZ7LN256, MAV0>
naa.5002538d417cc12c
sd1: 244198MB, 512 bytes/sector, 500118192 sectors, thin
ichiic0 at pci1 dev 31 function 3 "Intel C610 SMBus" rev 0x05: apic 1 int 18
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub3 at uhub0 port 12 configuration 1 interface 0 "ATEN International
product 0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub3 port 1 configuration 1 interface 0 "ATEN International
product 0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev1 at uhub3 port 1 configuration 1 interface 1 "ATEN International
product 0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
uhub5 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.05 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (06e397d0f983db15.a) swap on sd0b dump on sd0b
On 24.1.2024. 0:48, Hrvoje Popovski wrote:
> Hi all,
>
> in production I have simple carp pfsync setup with
> em0 - carp <- uplink
> em1 - pfsync
> ix0 - vlan - carp <- internal networks
> ix1 - not used
> and for vpn I have wireguard and people connects to em0 carp address.
> There's no bridges or tunnels or any exotic pf feature in this setup.
>
> Until this snapshot
> OpenBSD 7.4-current (GENERIC.MP) #1587: Sat Dec 30 22:44:51 MST 2023
> every this was fine,
> but with and after
> OpenBSD 7.4-current (GENERIC.MP) #1588: Thu Jan 4 20:58:35 MST 2024
> em0 starts to go up/down spontaneously and em0 watchdog logs start to
> appear in messages
>
> em0: watchdog: head 113 tail 178 TDH 178 TDT 113
> carp1: state transition: BACKUP -> MASTER
>
> even with net.inet.tcp.tso=0
>
>
> When reverting em TSO diffs if_em.c to r1.369 and if_em.h to r1.80
> firewall starts to work normally and em0 is fine.
>
> After rebooting firewall and promote it to carp master I've started to
> collect kstat em0::: after em0 watchdog log
>
>
> 1) Jan 22 08:01:01 fw2 /bsd: em0: watchdog: head 473 tail 25 TDH 25 TDT 473
> kstat em0::: - em0-1.txt
> 2) Jan 22 08:07:11 fw2 /bsd: em0: watchdog: head 114 tail 178 TDH 178
> TDT 114
> 3) Jan 22 08:08:16 fw2 /bsd: em0: watchdog: head 61 tail 126 TDH 126 TDT 61
> kstat em0::: - em0-3.txt
> 4) Jan 22 08:21:23 fw2 /bsd: em0: watchdog: head 452 tail 5 TDH 5 TDT 452
> 5) Jan 22 08:33:48 fw2 /bsd: em0: watchdog: head 352 tail 416 TDH 416
> TDT 352
> 6) Jan 22 08:36:20 fw2 /bsd: em0: watchdog: head 446 tail 510 TDH 510
> TDT 446
> kstat em0::: - em0-6.txt
> 7) Jan 22 08:42:16 fw1 /bsd: em0: watchdog: head 385 tail 450 TDH 450
> TDT 385
> kstat em0::: - em0-7.txt
>
>
> in the attachment you can find em0 txt kstat output and kstat-all.txt
> which is kstat of all interfaces with TSO diff after 7th time em0
> watchdog log
>
> From logs it seems that em0:0:txq:0 oactives counter, em0 watchdog and
> em0 going up/down is somehow connected because every time I see em0
> watchdog, oactives counter is increased by one
>
>
> log on switch
> I 01/22/24 08:01:01 00077 ports: port 2 is now off-line
> I 01/22/24 08:01:05 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:07:11 00077 ports: port 2 is now off-line
> I 01/22/24 08:07:14 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:08:16 00077 ports: port 2 is now off-line
> I 01/22/24 08:08:20 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:21:23 00077 ports: port 2 is now off-line
> I 01/22/24 08:21:26 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:33:47 00077 ports: port 2 is now off-line
> I 01/22/24 08:33:51 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:36:20 00077 ports: port 2 is now off-line
> I 01/22/24 08:36:24 00076 ports: port 2 is now on-line
>
> I 01/22/24 08:42:16 00077 ports: port 2 is now off-line
> I 01/22/24 08:42:20 00076 ports: port 2 is now on-line
>
> em0 is connected to port 2
> ix0 is connected to port 6 and it's up whole the time...
>
>
> Packet processing and some little pressure need to be over em0 to
> trigger em0 watchdog and only carp master is affected. Over night there
> are 2 or 3 em0 watchdogs. Firewalls are more than underutilized cca 5k
> states and under 100Mbps
>
> To rule out em hardware problem, I've sysupdate second firewall and
> problem was the same as on first one.
>
>
> I am willing to debug this further but I don't know what to look any
> more ...
>
> And of course, thank you guys for carp and pfsync, without it this would
> be a problem but it's not :)
>
>
>
>
> kstat em0::: after day without TSO diffs
>
> fw2# uptime
> 12:36AM up 1 day, 13:38, 2 users, load averages: 0.35, 0.23, 0.23
> fw2# kstat em0:::
> em0:0:em-stats:0
> rx crc errs: 0 packets
> rx align errs: 0 packets
> rx align errs: 0 packets
> rx errs: 0 packets
> rx missed: 0 packets
> tx single coll: 0 packets
> tx excess coll: 0 packets
> tx multi coll: 0 packets
> tx late coll: 0 packets
> tx coll: 0
> tx defers: 0
> tx no CRS: 0 packets
> seq errs: 0
> carr ext errs: 0 packets
> rx len errs: 0 packets
> rx xon: 0 packets
> tx xon: 0 packets
> rx xoff: 0 packets
> tx xoff: 0 packets
> FC unsupported: 0 packets
> rx 64B: 6361422 packets
> rx 65-127B: 19106140 packets
> rx 128-255B: 4430154 packets
> rx 256-511B: 5116503 packets
> rx 512-1023B: 7665843 packets
> rx 1024-maxB: 86778341 packets
> rx good: 129458403 packets
> rx bcast: 147 packets
> rx mcast: 112979 packets
> tx good: 67968976 packets
> rx good: 134077827262 bytes
> tx good: 32314161469 bytes
> rx no buffers: 4 packets
> rx undersize: 0 packets
> rx fragments: 0 packets
> rx oversize: 0 packets
> rx jabbers: 0 packets
> rx mgmt: 0 packets
> rx mgmt drops: 0 packets
> tx mgmt: 0 packets
> rx total: 134077827262 bytes
> tx total: 32314161469 bytes
> rx total: 129458403 packets
> tx total: 67968976 packets
> tx 64B: 8932448 packets
> tx 65-127B: 31092764 packets
> tx 128-255B: 3930861 packets
> tx 256-511B: 2126737 packets
> tx 512-1023B: 4009214 packets
> tx 1024-maxB: 17876952 packets
> tx mcast: 71830 packets
> tx bcast: 144 packets
> em0:0:rxq:0
> packets: 129458403 packets
> bytes: 133559993650 bytes
> fdrops: 0 packets
> qdrops: 0 packets
> errors: 0 packets
> qlen: 0 packets
> enqueues: 114415797
> dequeues: 114407008
> em0:0:txq:0
> packets: 67968978 packets
> bytes: 31989786911 bytes
> qdrops: 2 packets
> errors: 0 packets
> qlen: 0 packets
> maxqlen: 511 packets
> oactive: false
> oactives: 0
>
>
>
>
> OpenBSD 7.4-current (GENERIC.MP) #32: Mon Jan 22 10:57:01 CET 2024
> [email protected]:/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17052467200 (16262MB)
> avail mem = 16514764800 (15749MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (40 entries)
> bios0: vendor American Megatrends Inc. version "2.1" date 11/22/2019
> bios0: Supermicro Super Server
> acpi0 at bios0: ACPI 5.0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT
> SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
> acpi0: wakeup devices IP2P(S4) EHC1(S4) EHC2(S4) RP01(S4) RP02(S4)
> RP03(S4) RP04(S4) RP05(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4)
> BR2A(S4) BR2B(S4) BR2C(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.04 MHz, 06-56-03,
> patch 0700001c
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 100MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.07 MHz, 06-56-03,
> patch 0700001c
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.15 MHz, 06-56-03,
> patch 0700001c
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.20 MHz, 06-56-03,
> patch 0700001c
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 8 (application processor)
> cpu4: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.21 MHz, 06-56-03,
> patch 0700001c
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu4: smt 0, core 4, package 0
> cpu5 at mainbus0: apid 10 (application processor)
> cpu5: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.22 MHz, 06-56-03,
> patch 0700001c
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu5: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu5: smt 0, core 5, package 0
> cpu6 at mainbus0: apid 12 (application processor)
> cpu6: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.25 MHz, 06-56-03,
> patch 0700001c
> cpu6:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu6: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu6: smt 0, core 6, package 0
> cpu7 at mainbus0: apid 14 (application processor)
> cpu7: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.38 MHz, 06-56-03,
> patch 0700001c
> cpu7:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu7: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 12MB 64b/line 12-way L3 cache
> cpu7: smt 0, core 7, package 0
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0x80000000, bus 0-255
> acpihpet0 at acpi0: 14318179 Hz
> acpiprt0 at acpi0: bus 255 (UNC0)
> acpiprt1 at acpi0: bus 0 (PCI0)
> acpiprt2 at acpi0: bus 5 (RP01)
> acpiprt3 at acpi0: bus 6 (RP05)
> acpiprt4 at acpi0: bus 7 (ASTB)
> acpiprt5 at acpi0: bus 1 (BR1A)
> acpiprt6 at acpi0: bus 3 (BR2C)
> acpiprt7 at acpi0: bus 4 (BR3A)
> acpipci0 at acpi0 UNC0: 0x00000010 0x00000011 0x00000000
> "ACPI0004" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "PNP0C33" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> "PNP0003" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> acpicmos0 at acpi0
> com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
> com1: console
> "IPI0001" at acpi0 not configured
> acpibtn0 at acpi0: PWRB
> acpicpu0 at acpi0: C1(1000@1 halt), PSS
> acpicpu1 at acpi0: C1(1000@1 halt), PSS
> acpicpu2 at acpi0: C1(1000@1 halt), PSS
> acpicpu3 at acpi0: C1(1000@1 halt), PSS
> acpicpu4 at acpi0: C1(1000@1 halt), PSS
> acpicpu5 at acpi0: C1(1000@1 halt), PSS
> acpicpu6 at acpi0: C1(1000@1 halt), PSS
> acpicpu7 at acpi0: C1(1000@1 halt), PSS
> ipmi at mainbus0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> cpu0: Enhanced SpeedStep 2100 MHz: speeds: 2101, 2100, 2000, 1900, 1800,
> 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
> pci0 at mainbus0 bus 255
> 255:30:3: io address conflict 0x14/0x4
> "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 0 not configured
> "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 1 not configured
> "Intel Xeon-D QPI Link" rev 0x03 at pci0 dev 11 function 2 not configured
> "Intel Xeon-D QPI Debug" rev 0x03 at pci0 dev 11 function 3 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 0 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 1 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 2 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 12 function 3 not configured
> "Intel E5 v4 Cache" rev 0x03 at pci0 dev 12 function 4 not configured
> "Intel E5 v4 Cache" rev 0x03 at pci0 dev 12 function 5 not configured
> vendor "Intel", unknown product 0x6fe6 (class system subclass
> miscellaneous, rev 0x03) at pci0 dev 12 function 6 not configured
> vendor "Intel", unknown product 0x6fe7 (class system subclass
> miscellaneous, rev 0x03) at pci0 dev 12 function 7 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 0 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 4 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 5 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci0 dev 15 function 6 not configured
> "Intel Xeon-D PCIE" rev 0x03 at pci0 dev 16 function 0 not configured
> "Intel E5 v4 R2PCIe Agent" rev 0x03 at pci0 dev 16 function 1 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 5 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 6 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci0 dev 16 function 7 not configured
> "Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 0 not configured
> "Intel Xeon-D Home Agent" rev 0x03 at pci0 dev 18 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 3 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 4 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 5 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 6 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 19 function 7 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 3 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 4 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 5 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 6 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 20 function 7 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci0 dev 21 function 3 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 0 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 1 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 2 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 3 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 30 function 4 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 0 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci0 dev 31 function 2 not configured
> pci1 at mainbus0 bus 0
> pchb0 at pci1 dev 0 function 0 "Intel Xeon-D Host" rev 0x03
> ppb0 at pci1 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
> pci2 at ppb0 bus 1
> ppb1 at pci1 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x03
> pci3 at ppb1 bus 2
> "Intel Xeon-D QuickData" rev 0x00 at pci3 dev 0 function 0 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci3 dev 0 function 1 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci3 dev 0 function 2 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci3 dev 0 function 3 not configured
> ppb2 at pci1 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x03
> pci4 at ppb2 bus 3
> vendor "Intel", unknown product 0x10a6 (class undefined unknown subclass
> 0x00, rev 0x00) at pci4 dev 0 function 0 not configured
> ppb3 at pci1 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
> pci5 at ppb3 bus 4
> ix0 at pci5 dev 0 function 0 "Intel X540T" rev 0x01, msix, 8 queues,
> address a0:36:9f:29:f3:28
> ix1 at pci5 dev 0 function 1 "Intel X540T" rev 0x01, msix, 8 queues,
> address a0:36:9f:29:f3:2a
> "Intel Xeon-D Address Map" rev 0x03 at pci1 dev 5 function 0 not configured
> "Intel Xeon-D Hot Plug" rev 0x03 at pci1 dev 5 function 1 not configured
> "Intel Xeon-D RAS" rev 0x03 at pci1 dev 5 function 2 not configured
> xhci0 at pci1 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi,
> xHCI 1.0
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
> 3.00/1.00 addr 1
> "Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 0 not configured
> "Intel 8 Series MEI" rev 0x04 at pci1 dev 22 function 1 not configured
> ehci0 at pci1 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
> usb1 at ehci0: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> ppb4 at pci1 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> pci6 at ppb4 bus 5
> em0 at pci6 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> ac:1f:6b:14:bd:b2
> em1 at pci6 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> ac:1f:6b:14:bd:b3
> ppb5 at pci1 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> pci7 at ppb5 bus 6
> ppb6 at pci7 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
> pci8 at ppb6 bus 7
> vga1 at pci8 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
> wsdisplay at vga1 not configured
> ehci1 at pci1 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
> usb2 at ehci1: USB revision 2.0
> uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> pcib0 at pci1 dev 31 function 0 "Intel C224 LPC" rev 0x05
> ahci0 at pci1 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi,
> AHCI 1.3
> ahci0: port 0: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, MTFDDAV256TDL, M5M> naa.500a0751269794ce
> sd0: 244198MB, 512 bytes/sector, 500118192 sectors, thin
> ichiic0 at pci1 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic
> 8 int 18
> iic0 at ichiic0
> "Intel 8 Series Thermal" rev 0x05 at pci1 dev 31 function 6 not configured
> isa0 at pcib0
> isadma0 at isa0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> uhub3 at uhub0 port 3 configuration 1 interface 0 "Genesys Logic USB2.0
> Hub" rev 2.00/32.98 addr 2
> uhub4 at uhub0 port 4 configuration 1 interface 0 "ATEN International
> product 0x7000" rev 2.00/0.00 addr 3
> uhidev0 at uhub4 port 1 configuration 1 interface 0 "ATEN International
> product 0x2419" rev 1.10/1.00 addr 4
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0 mux 1
> uhidev1 at uhub4 port 1 configuration 1 interface 1 "ATEN International
> product 0x2419" rev 1.10/1.00 addr 4
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhub5 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.05 addr 2
> uhub6 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.05 addr 2
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (65d02aa7010489f5.a) swap on sd0b dump on sd0b
>
