Hi,
We are noticing a possible issue with the mcx driver in 6.6 stable.
We're using a Mellanox 25Gbs dual-port adapter, model "Mellanox ConnectX-4 Lx"
We have the mcx0+mcx1 in an aggr interface, and after a random amount of hours
the interfaces go into a broken state and then of course the aggr goes down.
When the issue appears, both mcx interfaces have constantly OACTIVE on and only
the Ofail counter from "netstat -I" is constantly increasing.
A ifconfig down/up on the mcx interfaces helps to recover and situation is back
to normal.
It's important to note that in our lab we have exactly the same setup and the
issue does not appear there.
The lab has far less traffic, both in variety and volume. This leads me to
believe that something in the traffic is causing this.
Here are the ifconfigs and dmesg.
Thanks.
mcx0: flags=8d43<UP,BROADCAST,RUNNING,PROMISC,OACTIVE,SIMPLEX,MULTICAST> mtu
1500
lladdr fe:e1:ba:d0:52:57
description: slx-ixn-r1a
index 1 priority 0 llprio 3
trunk: trunkdev aggr0
media: Ethernet autoselect (25GbaseCR)
status: active
aggr0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
lladdr fe:e1:ba:d0:52:57
description: slx-ixn-r1
index 9 priority 0 llprio 7
trunk: trunkproto lacp
trunk id: [(8000,fe:e1:ba:d0:52:57,0009,0000,0000),
(0000,00:00:00:00:00:00,0000,0000,0000)]
mcx0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:52:57, key
0x9, port pri 0x8000 number 0x1
mcx0 lacp actor state activity,timeout,aggregation,defaulted
mcx0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key
0x0, port pri 0x0 number 0x0
mcx0 lacp partner state activity,timeout,aggregation,sync
mcx0 port
mcx1 lacp actor system pri 0x8000 mac fe:e1:ba:d0:52:57, key
0x9, port pri 0x8000 number 0x2
mcx1 lacp actor state activity,timeout,aggregation,defaulted
mcx1 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key
0x0, port pri 0x0 number 0x0
mcx1 lacp partner state activity,timeout,aggregation,sync
mcx1 port
groups: aggr
media: Ethernet autoselect
status: no carrier
OpenBSD 6.6 (GENERIC.MP) #4: Wed Jan 15 10:55:43 MST 2020
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17031753728 (16242MB)
avail mem = 16502853632 (15738MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x79173000 (241 entries)
bios0: vendor HP version "P89" date 05/21/2018
bios0: HP ProLiant DL360 Gen9
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP UEFI MCEJ SSDT HEST BERT ERST EINJ HPET PMCT WDDT APIC
MCFG SLIT SRAT SPMI RASF SPCR MSCT BDAT PCCT SSDT SSDT SSDT DMAR
acpi0: wakeup devices PEX4(S4) BR05(S4) BR03(S4) BR07(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 2097.92 MHz, 06-4f-01
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 2097.59 MHz, 06-4f-01
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 2, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 2097.59 MHz, 06-4f-01
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz, 2097.59 MHz, 06-4f-01
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0x80000000, bus 0-255
acpiprt0 at acpi0: bus 255 (UNC0)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 1 (PEX2)
acpiprt3 at acpi0: bus 2 (PEX4)
acpiprt4 at acpi0: bus 3 (BR01)
acpiprt5 at acpi0: bus 4 (BR05)
acpiprt6 at acpi0: bus 5 (BR03)
acpiprt7 at acpi0: bus 8 (BR07)
acpicpu0 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu1 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu2 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
acpicpu3 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1)
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpipci0 at acpi0 UNC0: _OSC failed
"ACPI0004" at acpi0 not configured
acpipci1 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"ACPI000E" at acpi0 not configured
"ACPI000D" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon-D Host" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x01
pci1 at ppb0 bus 3
ciss0 at pci1 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x01: apic 8
int 16
ciss0: 1 LD, HW rev 1, FW 6.60/6.60, 64bit fifo rro
scsibus1 at ciss0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <HP, LOGICAL VOLUME, 6.60>
sd0: 139979MB, 512 bytes/sector, 286677120 sectors
ppb1 at pci0 dev 1 function 1 "Intel Xeon-D PCIE" rev 0x01
pci2 at ppb1 bus 12
ppb2 at pci0 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x01: msi
pci3 at ppb2 bus 5
ppb3 at pci0 dev 2 function 1 "Intel Xeon-D PCIE" rev 0x01
pci4 at ppb3 bus 13
ppb4 at pci0 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x01
pci5 at ppb4 bus 4
ppb5 at pci0 dev 2 function 3 "Intel Xeon-D PCIE" rev 0x01
pci6 at ppb5 bus 14
ppb6 at pci0 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x01: msi
pci7 at ppb6 bus 8
mcx0 at pci7 dev 0 function 0 "Mellanox ConnectX-4 Lx" rev 0x00: FW 14.25.1020,
msix, address 1c:34:da:64:65:48
mcx1 at pci7 dev 0 function 1 "Mellanox ConnectX-4 Lx" rev 0x00: FW 14.25.1020,
msix, address 1c:34:da:64:65:49
ppb7 at pci0 dev 3 function 1 "Intel Xeon-D PCIE" rev 0x01
pci8 at ppb7 bus 15
ppb8 at pci0 dev 3 function 2 "Intel Xeon-D PCIE" rev 0x01
pci9 at ppb8 bus 11
ppb9 at pci0 dev 3 function 3 "Intel Xeon-D PCIE" rev 0x01
pci10 at ppb9 bus 16
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 0 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 1 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 2 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 3 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 4 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 5 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 6 not configured
"Intel E5 v4 DMA" rev 0x01 at pci0 dev 4 function 7 not configured
"Intel Xeon-D Address Map" rev 0x01 at pci0 dev 5 function 0 not configured
"Intel Xeon-D Hot Plug" rev 0x01 at pci0 dev 5 function 1 not configured
"Intel Xeon-D RAS" rev 0x01 at pci0 dev 5 function 2 not configured
"Intel Xeon-D I/O APIC" rev 0x01 at pci0 dev 5 function 4 not configured
"Intel C610 MS SPSR" rev 0x05 at pci0 dev 17 function 0 not configured
ehci0 at pci0 dev 26 function 0 "Intel C610 USB" rev 0x05: apic 8 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
ppb10 at pci0 dev 28 function 0 "Intel C610 PCIE" rev 0xd5
pci11 at ppb10 bus 17
ppb11 at pci0 dev 28 function 2 "Intel C610 PCIE" rev 0xd5
pci12 at ppb11 bus 1
"Hewlett-Packard iLO3 Slave" rev 0x06 at pci12 dev 0 function 0 not configured
vga1 at pci12 dev 0 function 1 "Matrox MGA G200eH" rev 0x01
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Hewlett-Packard iLO3 Management" rev 0x06 at pci12 dev 0 function 2 not
configured
uhci0 at pci12 dev 0 function 4 "Hewlett-Packard USB" rev 0x03: apic 8 int 19
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Hewlett-Packard UHCI root hub" rev
1.00/1.00 addr 1
ppb12 at pci0 dev 28 function 4 "Intel C610 PCIE" rev 0xd5
pci13 at ppb12 bus 2
bge0 at pci13 dev 0 function 0 "Broadcom BCM5719" rev 0x01, unknown BCM5719
(0x5719001), APE firmware NCSI 1.4.22.0: msi, address 30:e1:71:6c:33:20
brgphy0 at bge0 phy 1: BCM5719C 10/100/1000baseT PHY, rev. 0
bge1 at pci13 dev 0 function 1 "Broadcom BCM5719" rev 0x01, unknown BCM5719
(0x5719001), APE firmware NCSI 1.4.22.0: msi, address 30:e1:71:6c:33:21
brgphy1 at bge1 phy 2: BCM5719C 10/100/1000baseT PHY, rev. 0
bge2 at pci13 dev 0 function 2 "Broadcom BCM5719" rev 0x01, unknown BCM5719
(0x5719001), APE firmware NCSI 1.4.22.0: msi, address 30:e1:71:6c:33:22
brgphy2 at bge2 phy 3: BCM5719C 10/100/1000baseT PHY, rev. 0
bge3 at pci13 dev 0 function 3 "Broadcom BCM5719" rev 0x01, unknown BCM5719
(0x5719001), APE firmware NCSI 1.4.22.0: msi, address 30:e1:71:6c:33:23
brgphy3 at bge3 phy 4: BCM5719C 10/100/1000baseT PHY, rev. 0
ppb13 at pci0 dev 28 function 6 "Intel C610 PCIE" rev 0xd5
pci14 at ppb13 bus 18
ppb14 at pci0 dev 28 function 7 "Intel C610 PCIE" rev 0xd5
pci15 at ppb14 bus 19
ehci1 at pci0 dev 29 function 0 "Intel C610 USB" rev 0x05: apic 8 int 18
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
pcib0 at pci0 dev 31 function 0 "Intel C610 LPC" rev 0x05
ahci0 at pci0 dev 31 function 2 "Intel C610 AHCI" rev 0x05: msi, AHCI 1.3
ahci0: port 5: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 5 lun 0: <hp, DVDRW GUD0N, UM01> removable
ichiic0 at pci0 dev 31 function 3 "Intel C610 SMBus" rev 0x05: apic 8 int 18
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: probed fifo depth: 0 bytes
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
pci16 at mainbus0 bus 255
"Intel Xeon-D QPI Link" rev 0x01 at pci16 dev 11 function 0 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci16 dev 11 function 1 not configured
"Intel Xeon-D QPI Link" rev 0x01 at pci16 dev 11 function 2 not configured
"Intel Xeon-D QPI Debug" rev 0x01 at pci16 dev 11 function 3 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 12 function 0 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 12 function 1 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 12 function 2 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 12 function 3 not configured
"Intel E5 v4 Cache" rev 0x01 at pci16 dev 12 function 4 not configured
"Intel E5 v4 Cache" rev 0x01 at pci16 dev 12 function 5 not configured
vendor "Intel", unknown product 0x6fe6 (class system subclass miscellaneous,
rev 0x01) at pci16 dev 12 function 6 not configured
vendor "Intel", unknown product 0x6fe7 (class system subclass miscellaneous,
rev 0x01) at pci16 dev 12 function 7 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 15 function 0 not configured
"Intel E5 v4 Cache" rev 0x01 at pci16 dev 15 function 1 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 15 function 4 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 15 function 5 not configured
"Intel Xeon-D Cache" rev 0x01 at pci16 dev 15 function 6 not configured
"Intel Xeon-D PCIE" rev 0x01 at pci16 dev 16 function 0 not configured
"Intel E5 v4 R2PCIe Agent" rev 0x01 at pci16 dev 16 function 1 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci16 dev 16 function 5 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci16 dev 16 function 6 not configured
"Intel Xeon-D Ubox" rev 0x01 at pci16 dev 16 function 7 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci16 dev 18 function 0 not configured
"Intel Xeon-D Home Agent" rev 0x01 at pci16 dev 18 function 1 not configured
vendor "Intel", unknown product 0x6f70 (class system subclass miscellaneous,
rev 0x01) at pci16 dev 18 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 4 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 5 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 19 function 7 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 3 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 4 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 5 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 6 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 20 function 7 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 21 function 0 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 21 function 1 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 21 function 2 not configured
"Intel Xeon-D Memory" rev 0x01 at pci16 dev 21 function 3 not configured
"Intel E5 v4 RAS" rev 0x01 at pci16 dev 22 function 0 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 22 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 22 function 7 not configured
"Intel E5 v4 Thermal" rev 0x01 at pci16 dev 23 function 0 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 23 function 4 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 23 function 5 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 23 function 6 not configured
"Intel E5 v4 DDRIO" rev 0x01 at pci16 dev 23 function 7 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 30 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 30 function 1 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 30 function 2 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 30 function 3 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 30 function 4 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 31 function 0 not configured
"Intel Xeon-D PCU" rev 0x01 at pci16 dev 31 function 2 not configured
vmm0 at mainbus0: VMX/EPT
uhub3 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.05 addr 2
uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.05 addr 2
uhub5 at uhub4 port 3 configuration 1 interface 0 "Standard Microsystems
product 0x2660" rev 2.00/8.01 addr 3
uhidev0 at uhub1 port 1 configuration 1 interface 0 "BMC Virtual Keyboard" rev
1.10/0.02 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "BMC Virtual Keyboard" rev
1.10/0.02 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (dedd1e9a4177b857.a) swap on sd0b dump on sd0b
