It seems this lock order issue is not parallel diff specific.
> On 12 May 2021, at 12:58, Hrvoje Popovski <hrv...@srce.hr> wrote:
>
> On 21.4.2021. 21:36, Alexander Bluhm wrote:
>> We need more MP preassure to find such bugs and races. I think now
>> is a good time to give this diff broader testing and commit it.
>> You need interfaces with multiple queues to see a difference.
>
> Hi,
>
> while forwarding ip4 traffic over box with parallel diff and aggr
> interfaces, and then aggr is destroyed, i'm getting witness log below...
>
> i can't reproduce this log without parallel diff and i'm getting this
> log only first time destroying aggr interface ..
>
>
>
> witness: lock order reversal:
> 1st 0xffffffff82139de0 netlock (netlock)
> 2nd 0xffffffff82120bb8 timeout (timeout)
> lock order "timeout"(rwlock) -> "netlock"(rwlock) first seen at:
> #0 rw_enter_write+0x43
> #1 mld6_fasttimeo+0x14
> #2 pffasttimo+0x67
> #3 timeout_run+0x93
> #4 softclock_thread+0x11d
> #5 proc_trampoline+0x1c
> lock order "netlock"(rwlock) -> "timeout"(rwlock) first seen at:
> #0 timeout_del_barrier+0x41
> #1 aggr_p_dtor+0x17b
> #2 aggr_clone_destroy+0x91
> #3 if_clone_destroy+0xd8
> #4 ifioctl+0x1d2
> #5 soo_ioctl+0x171
> #6 sys_ioctl+0x2c4
> #7 syscall+0x3b9
> #8 Xsyscall+0x128
>
>
> r620-1# cat /etc/hostname.aggr0
> trunkport ix0
> lladdr ec:f4:bb:da:f7:f8
> inet 192.168.42.1 255.255.255.0
> !route add 16/8 192.168.42.11
> up
>
> r620-1# cat /etc/hostname.aggr1
> trunkport ix1
> inet 192.168.43.1 255.255.255.0
> !route add 48/8 192.168.43.11
> up
>
>
>
> OpenBSD 6.9-current (GENERIC.MP) #166: Wed May 12 10:18:11 CEST 2021
> hrv...@r620-1.srce.hr:/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17115840512 (16322MB)
> avail mem = 16450007040 (15687MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
> bios0: vendor Dell Inc. version "2.9.0" date 12/06/2019
> bios0: Dell Inc. PowerEdge R620
> acpi0 at bios0: ACPI 3.0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WDAT SLIC ERST HEST
> BERT EINJ TCPA PC__ SRAT SSDT
> acpi0: wakeup devices PCI0(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 4 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.44 MHz, 06-3e-04
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 2, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 6 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.04 MHz, 06-3e-04
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 3, package 0
> cpu2 at mainbus0: apid 8 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 4, package 0
> cpu3 at mainbus0: apid 16 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 8, package 0
> cpu4 at mainbus0: apid 18 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 0, core 9, package 0
> cpu5 at mainbus0: apid 20 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.03 MHz, 06-3e-04
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 0, core 10, package 0
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped
> acpihpet0 at acpi0: 14318179 Hz
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (PEX1)
> acpiprt2 at acpi0: bus -1 (PE1C)
> acpiprt3 at acpi0: bus 3 (PEX2)
> acpiprt4 at acpi0: bus 2 (PEX3)
> acpiprt5 at acpi0: bus 4 (PEX4)
> acpiprt6 at acpi0: bus -1 (PEX5)
> acpiprt7 at acpi0: bus 8 (PEX6)
> acpiprt8 at acpi0: bus 7 (PEX7)
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpicmos0 at acpi0
> acpipci1 at acpi0 P0B1: 0x00000000 0x00000011 0x00000001
> acpipci2 at acpi0 P1B1: 0x00000000 0x00000011 0x00000001
> "PNP0C14" at acpi0 not configured
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> acpicpu2 at acpi0: C1(@1 halt!)
> acpicpu3 at acpi0: C1(@1 halt!)
> acpicpu4 at acpi0: C1(@1 halt!)
> acpicpu5 at acpi0: C1(@1 halt!)
> ipmi at mainbus0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel E5 v2 Host" rev 0x04
> ppb0 at pci0 dev 1 function 0 "Intel E5 v2 PCIE" rev 0x04
> pci1 at ppb0 bus 1
> 1:0:1: rom address conflict 0xd8000000/0x80000
> ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues,
> address ec:f4:bb:da:f7:f8
> ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues,
> address ec:f4:bb:da:f7:fa
> ppb1 at pci0 dev 2 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
> pci2 at ppb1 bus 3
> ppb2 at pci0 dev 2 function 2 "Intel E5 v2 PCIE" rev 0x04
> pci3 at ppb2 bus 2
> mfi0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
> apic 1 int 10
> mfi0: "PERC H310 Mini", firmware 20.13.3-0001
> scsibus1 at mfi0: 16 targets
> sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H310, 2.12>
> naa.6c81f660f14398001af3a49619e0d82f
> sd0: 285568MB, 512 bytes/sector, 584843264 sectors
> scsibus2 at mfi0: 256 targets
> ppb3 at pci0 dev 3 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
> pci4 at ppb3 bus 4
> 4:0:1: rom address conflict 0xda000000/0x80000
> ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 0, FW
> 8.2.64244 API 1.13, msix, 4 queues, address 3c:fd:fe:04:0d:64
> ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 1, FW
> 8.2.64244 API 1.13, msix, 4 queues, address 3c:fd:fe:04:0d:66
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 0 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 1 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 2 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 3 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 4 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 5 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 6 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 7 not configured
> "Intel E5 v2 Address Map" rev 0x04 at pci0 dev 5 function 0 not configured
> "Intel E5 v2 IIO RAS" rev 0x04 at pci0 dev 5 function 2 not configured
> ppb4 at pci0 dev 17 function 0 "Intel C600 Virtual PCIE" rev 0x05
> pci5 at ppb4 bus 5
> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
> ehci0 at pci0 dev 26 function 0 "Intel C600 USB" rev 0x05: apic 0 int 23
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> ppb5 at pci0 dev 28 function 0 "Intel C600 PCIE" rev 0xb5
> pci6 at ppb5 bus 6
> ppb6 at pci0 dev 28 function 4 "Intel C600 PCIE" rev 0xb5
> pci7 at ppb6 bus 7
> em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> ec:f4:bb:da:f7:fc
> em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> ec:f4:bb:da:f7:fd
> ppb7 at pci0 dev 28 function 7 "Intel C600 PCIE" rev 0xb5
> pci8 at ppb7 bus 8
> ppb8 at pci8 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci9 at ppb8 bus 9
> ppb9 at pci9 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci10 at ppb9 bus 10
> ppb10 at pci10 dev 0 function 0 "Renesas SH7757 PCIE-PCI" rev 0x00
> pci11 at ppb10 bus 11
> vga1 at pci11 dev 0 function 0 "Matrox MGA G200eR" rev 0x00
> wsdisplay at vga1 not configured
> ppb11 at pci9 dev 1 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci12 at ppb11 bus 12
> ehci1 at pci0 dev 29 function 0 "Intel C600 USB" rev 0x05: apic 0 int 22
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> ppb12 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
> pci13 at ppb12 bus 13
> pcib0 at pci0 dev 31 function 0 "Intel C600 LPC" rev 0x05
> ahci0 at pci0 dev 31 function 2 "Intel C600 AHCI" rev 0x05: msi, AHCI 1.3
> ahci0: port 4: 1.5Gb/s
> scsibus3 at ahci0: 32 targets
> cd0 at scsibus3 targ 4 lun 0: <TSSTcorp, DVD+-RW SU-208FB, D150> removable
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> com1: console
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> pci14 at mainbus0 bus 63
> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 8 function 0 not configured
> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 9 function 0 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 0 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 1 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 2 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 3 not configured
> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 0 not configured
> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 0 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 1 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 2 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 4 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 0 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 1 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 2 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 4 not configured
> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 0 not configured
> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 1 not configured
> "Intel E5 v2 TA" rev 0x04 at pci14 dev 15 function 0 not configured
> "Intel E5 v2 RAS" rev 0x04 at pci14 dev 15 function 1 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 2 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 3 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 4 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 5 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 0 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 1 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 2 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 3 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 4 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 5 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 7 not configured
> "Intel E5 v2 R2PCIE" rev 0x04 at pci14 dev 19 function 0 not configured
> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 1 not
> configured
> "Intel E5 v2 QPI" rev 0x04 at pci14 dev 19 function 4 not configured
> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 5 not
> configured
> "Intel E5 v2 SAD" rev 0x04 at pci14 dev 22 function 0 not configured
> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 1 not configured
> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 2 not configured
> pci15 at mainbus0 bus 127
> vmm0 at mainbus0: VMX/EPT
> dt: 443 probes
> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.00 addr 2
> uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
> Gadget USB HUB" rev 2.00/0.00 addr 3
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0 mux 1
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev2: iclass 3/1
> ums1 at uhidev2: 3 buttons, Z dir
> wsmouse1 at ums1 mux 0
> uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.00 addr 2
> vscsi0 at root
> scsibus4 at vscsi0: 256 targets
> softraid0 at root
> scsibus5 at softraid0: 256 targets
> root on sd0a (38afb4d76f75540e.a) swap on sd0b dump on sd0b
> wskbd0 detached
> ukbd0 detached
> uhidev0 detached
> wsmouse0 detached
> ums0 detached
> uhidev1 detached
> wsmouse1 detached
> ums1 detached
> uhidev2 detached
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0 mux 1
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev2: iclass 3/1
> ums1 at uhidev2: 3 buttons, Z dir
> wsmouse1 at ums1 mux 0
> witness: lock order reversal:
> 1st 0xffffffff82139de0 netlock (netlock)
> 2nd 0xffffffff82120bb8 timeout (timeout)
> lock order "timeout"(rwlock) -> "netlock"(rwlock) first seen at:
> #0 rw_enter_write+0x43
> #1 mld6_fasttimeo+0x14
> #2 pffasttimo+0x67
> #3 timeout_run+0x93
> #4 softclock_thread+0x11d
> #5 proc_trampoline+0x1c
> lock order "netlock"(rwlock) -> "timeout"(rwlock) first seen at:
> #0 timeout_del_barrier+0x41
> #1 aggr_p_dtor+0x17b
> #2 aggr_clone_destroy+0x91
> #3 if_clone_destroy+0xd8
> #4 ifioctl+0x1d2
> #5 soo_ioctl+0x171
> #6 sys_ioctl+0x2c4
> #7 syscall+0x3b9
> #8 Xsyscall+0x128
> em1: watchdog: head 1 tail 0 TDH 1 TDT 1
> em1: watchdog: head 1 tail 0 TDH 1 TDT 1
> em1: watchdog: head 1 tail 0 TDH 1 TDT 1
> em1: watchdog: head 1 tail 0 TDH 1 TDT 1
> r620-1#
>
