Re: Witness lock-order reversal in radeondrm

2022-04-27 Thread Mark Kettenis
> Date: Wed, 27 Apr 2022 13:52:28 -0400 (EDT)
> From: d...@sisu.io
> 
> >Synopsis:Witnesss lock order reversal in radeondrm
> >Category:kernel
> >Environment:
>   System  : OpenBSD 7.1
>   Details : OpenBSD 7.1-current (CUSTOM.MP) #14: Wed Apr 27 13:22:39 
> EDT 2022
>
> d...@minmin.sisu.home:/usr/src/sys/arch/amd64/compile/CUSTOM.MP
> 
>   Architecture: OpenBSD.amd64
>   Machine : amd64
> >Description:
> 
> Noticed this on a fresh kernel I built while hacking on some vmm/vmd
> stuff. Probably related to the MP_LOCKDEBUG spinout I reported
> previously and since bumped the spinout counter to INT_MAX.

I doubt it.

The missing lock order data means we can't know for sure, but since
pretty much all of the drm code still runs under the kernel lock, lock
order reversals aren't necessarily problematic.

> The following was in my kernel buffer after rebooting and logging
> into my box:
> 
> witness: lock order reversal:
>  1st 0xfd91bffda1d0 uobjlk (>vmobjlock)
>  2nd 0x80430b78 mclk (>pm.mclk_lock)
> lock order data w2 -> w1 missing
> lock order ">vmobjlock"(rwlock) -> ">pm.mclk_lock"(rwlock) first 
> seen at:
> #0  rw_enter_read+0x38
> #1  radeon_gem_fault+0x4e
> #2  uvm_fault+0x179
> #3  upageflttrap+0x62
> #4  usertrap+0x129
> #5  recall_trap+0x8
> 
> >How-To-Repeat:
> My kernel config:
> 
> include "arch/amd64/conf/GENERIC"
> 
> #option   VMM_DEBUG
> optionMULTIPROCESSOR
> optionMP_LOCKDEBUG
> optionWITNESS
> 
> cpu*  at mainbus?
> 
> >Fix:
>  TBD
> 
> -dv
> 
> dmesg:
> OpenBSD 7.1-current (CUSTOM.MP) #14: Wed Apr 27 13:22:39 EDT 2022
> d...@minmin.sisu.home:/usr/src/sys/arch/amd64/compile/CUSTOM.MP
> real mem = 85769121792 (81795MB)
> avail mem = 82520170496 (78697MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec0f0 (105 entries)
> bios0: vendor Dell Inc. version "A32" date 09/25/2019
> bios0: Dell Inc. Precision Tower 7810
> acpi0 at bios0: ACPI 5.0
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP APIC FPDT FIDT MCFG UEFI HPET MSCT SLIT SRAT SRAT 
> WDDT SSDT NITR SLIC MSDM DMAR ASF!
> acpi0: wakeup devices IP2P(S3) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP06(S4) 
> RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) BR2D(S4) 
> BR3A(S4) BR3B(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.63 MHz, 06-3f-02
> cpu0: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
> cpu1: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: disabling user TSC (skew=9173028)
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
> cpu2: 
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: disabling user TSC (skew=9880086)
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
> cpu3: 
> 

Witness lock-order reversal in radeondrm

2022-04-27 Thread dave
>Synopsis:  Witnesss lock order reversal in radeondrm
>Category:  kernel
>Environment:
System  : OpenBSD 7.1
Details : OpenBSD 7.1-current (CUSTOM.MP) #14: Wed Apr 27 13:22:39 
EDT 2022
 
d...@minmin.sisu.home:/usr/src/sys/arch/amd64/compile/CUSTOM.MP

Architecture: OpenBSD.amd64
Machine : amd64
>Description:

Noticed this on a fresh kernel I built while hacking on some vmm/vmd
stuff. Probably related to the MP_LOCKDEBUG spinout I reported
previously and since bumped the spinout counter to INT_MAX.

The following was in my kernel buffer after rebooting and logging
into my box:

witness: lock order reversal:
 1st 0xfd91bffda1d0 uobjlk (>vmobjlock)
 2nd 0x80430b78 mclk (>pm.mclk_lock)
lock order data w2 -> w1 missing
lock order ">vmobjlock"(rwlock) -> ">pm.mclk_lock"(rwlock) first 
seen at:
#0  rw_enter_read+0x38
#1  radeon_gem_fault+0x4e
#2  uvm_fault+0x179
#3  upageflttrap+0x62
#4  usertrap+0x129
#5  recall_trap+0x8

>How-To-Repeat:
My kernel config:

include "arch/amd64/conf/GENERIC"

#option VMM_DEBUG
option  MULTIPROCESSOR
option  MP_LOCKDEBUG
option  WITNESS

cpu*at mainbus?

>Fix:
 TBD

-dv

dmesg:
OpenBSD 7.1-current (CUSTOM.MP) #14: Wed Apr 27 13:22:39 EDT 2022
d...@minmin.sisu.home:/usr/src/sys/arch/amd64/compile/CUSTOM.MP
real mem = 85769121792 (81795MB)
avail mem = 82520170496 (78697MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec0f0 (105 entries)
bios0: vendor Dell Inc. version "A32" date 09/25/2019
bios0: Dell Inc. Precision Tower 7810
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT MCFG UEFI HPET MSCT SLIT SRAT SRAT WDDT 
SSDT NITR SLIC MSDM DMAR ASF!
acpi0: wakeup devices IP2P(S3) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP06(S4) 
RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) BR2C(S4) BR2D(S4) 
BR3A(S4) BR3B(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.63 MHz, 06-3f-02
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: disabling user TSC (skew=9173028)
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: disabling user TSC (skew=9880086)
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz, 3392.16 MHz, 06-3f-02
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: disabling user TSC (skew=8299192)
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-2643 v3 @