Hey, Yeah I remember hitting one of those quite often a few weeks ago with witness(4) enabled, a LOR between the inode lock and the vmmap lock. My initial guess was that this was a false positive but the small amount of time I dedicated to it wasn't enough to prove or disprove it...
Mathieu. David Hill wrote: > Hello - > > I enabled witness(4) and vfs lock in hopes to discover a hardlock after > Oct 29. I drop into the debugger on boot when nsd(8) starts: > > OpenBSD 6.2-current (GENERIC.MP) #37: Thu Nov 9 09:52:26 EST 2017 > [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 17032458240 (16243MB) > avail mem = 16378048512 (15619MB) > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb450 (75 entries) > bios0: vendor American Megatrends Inc. version "F15" date 10/23/2013 > bios0: Gigabyte Technology Co., Ltd. B75M-D3H > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S3 S4 S5 > acpi0: tables DSDT FACP APIC MCFG HPET SSDT SSDT SSDT DMAR > acpi0: wakeup devices PS2K(S3) P0P1(S4) USB1(S3) USB2(S3) USB3(S3) USB4(S3) > USB5(S3) USB6(S3) USB7(S3) PXSX(S4) RP01( > S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.78 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT > ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN > E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu0: 256KB 64b/line 8-way L2 cache > acpitimer0: recalibrated TSC frequency 3392291239 Hz > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges > cpu0: apic clock running at 99MHz > cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT > ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN > E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: smt 0, core 1, package 0 > cpu2 at mainbus0: apid 4 (application processor) > cpu2: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu2: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT > ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu2: 256KB 64b/line 8-way L2 cache > cpu2: smt 0, core 2, package 0 > cpu3 at mainbus0: apid 6 (application processor) > cpu3: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu3: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu3: 256KB 64b/line 8-way L2 cache > cpu3: smt 0, core 3, package 0 > cpu4 at mainbus0: apid 1 (application processor) > cpu4: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu4: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu4: 256KB 64b/line 8-way L2 cache > cpu4: smt 1, core 0, package 0 > cpu5 at mainbus0: apid 3 (application processor) > cpu5: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu5: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu5: 256KB 64b/line 8-way L2 cache > cpu5: smt 1, core 1, package 0 > cpu6 at mainbus0: apid 5 (application processor) > cpu6: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu6: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu6: 256KB 64b/line 8-way L2 cache > cpu6: smt 1, core 2, package 0 > cpu7 at mainbus0: apid 7 (application processor) > cpu7: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz > cpu7: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT > cpu7: 256KB 64b/line 8-way L2 cache > cpu7: smt 1, core 3, package 0 > ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins > acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63 > acpihpet0 at acpi0: 14318179 Hz > acpihpet0: recalibrated TSC frequency 3392306800 Hz > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 5 (P0P1) > acpiprt2 at acpi0: bus 1 (RP01) > acpiprt3 at acpi0: bus -1 (RP02) > acpiprt4 at acpi0: bus -1 (RP03) > acpiprt5 at acpi0: bus -1 (RP04) > acpiprt6 at acpi0: bus 4 (RP05) > acpiprt7 at acpi0: bus -1 (RP06) > acpiprt8 at acpi0: bus -1 (RP07) > acpiprt9 at acpi0: bus -1 (RP08) > acpiprt10 at acpi0: bus -1 (PEG0) > acpiprt11 at acpi0: bus -1 (PEG1) > acpiprt12 at acpi0: bus -1 (PEG2) > acpiprt13 at acpi0: bus -1 (PEG3) > acpiec0 at acpi0: not present > acpicpu0 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu1 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu2 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu3 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu4 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu5 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu6 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpicpu7 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS > acpipwrres0 at acpi0: FN00, resource for FAN0 > acpipwrres1 at acpi0: FN01, resource for FAN1 > acpipwrres2 at acpi0: FN02, resource for FAN2 > acpipwrres3 at acpi0: FN03, resource for FAN3 > acpipwrres4 at acpi0: FN04, resource for FAN4 > acpitz0 at acpi0: critical temperature is 106 degC > "INT3F0D" at acpi0 not configured > "PNP0400" at acpi0 not configured > acpibtn0 at acpi0: PWRB > "PNP0C0B" at acpi0 not configured > "PNP0C0B" at acpi0 not configured > "PNP0C0B" at acpi0 not configured > "PNP0C0B" at acpi0 not configured > "PNP0C0B" at acpi0 not configured > acpivideo0 at acpi0: GFX0 > acpivout0 at acpivideo0: DD02 > cpu0: Enhanced SpeedStep 3392 MHz: speeds: 3901, 3900, 3700, 3600, 3400, > 3200, 3100, 2900, 2700, 2600, 2400, 2300, 2100, 1900, 1800, 1600 MHz > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09 > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09 > drm0 at inteldrm0 > inteldrm0: msi > inteldrm0: 1280x1024, 32bpp > wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation) > wsdisplay0: screen 1-5 added (std, vt100 emulation) > xhci0 at pci0 dev 20 function 0 "Intel 7 Series xHCI" rev 0x04: msi > usb0 at xhci0: USB revision 3.0 > uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 > addr 1 > "Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured > ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 16 > usb1 at ehci0: USB revision 2.0 > uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 > addr 1 > ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi > pci1 at ppb0 bus 1 > ppb1 at pci1 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00 > pci2 at ppb1 bus 2 > mfi0 at pci2 dev 14 function 0 "Symbios Logic MegaRAID SAS 1064R" rev 0x00: > apic 2 int 18 > mfi0: "MegaRAID SAS 84016E", firmware 7.0.1-0084, 256MB cache > scsibus1 at mfi0: 64 targets > sd0 at scsibus1 targ 0 lun 0: <LSI, MegaRAID 84016E, 1.12> SCSI3 0/direct > fixed naa.600605b001d5a4101c162d40411f21f2 > sd0: 2858160MB, 512 bytes/sector, 5853511680 sectors > ppb2 at pci1 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00 > pci3 at ppb2 bus 3 > ppb3 at pci0 dev 28 function 4 "Intel 7 Series PCIE" rev 0xc4: msi > pci4 at ppb3 bus 4 > re0 at pci4 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL > (0x2c80), msi, address fc:aa:14:29:1d:1d > rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 5 > ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 23 > usb2 at ehci1: USB revision 2.0 > uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 > addr 1 > ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4 > pci5 at ppb4 bus 5 > fxp0 at pci5 dev 1 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 19, > address 00:0e:0c:35:70:48 > inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4 > pcib0 at pci0 dev 31 function 0 "Intel B75 LPC" rev 0x04 > ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3 > ahci0: port 0: 6.0Gb/s > scsibus2 at ahci0: 32 targets > sd1 at scsibus2 targ 0 lun 0: <ATA, Crucial_CT128MX1, MU01> SCSI3 0/direct > fixed naa.500a07510d04dca2 > sd1: 122104MB, 512 bytes/sector, 250069680 sectors, thin > ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int > 18 > iic0 at ichiic0 > spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800 > spdmem1 at iic0 addr 0x53: 8GB DDR3 SDRAM PC3-12800 > isa0 at pcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pckbd0 at pckbc0 (kbd slot) > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > lpt0 at isa0 port 0x378/4 irq 7 > it0 at isa0 port 0x2e/2: IT8728F rev 1, EC port 0xa30 > error: [drm:pid0:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo > underrun on pch transcoder A > error: [drm:pid0:intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder > A FIFO underrun > uhidev0 at uhub0 port 2 configuration 1 interface 0 "American Power > Conversion Back-UPS RS 700G FW:856.L5 -P.D USB FW:L5 -P" rev 2.00/0.90 addr 2 > uhidev0: iclass 3/0, 146 report ids > upd0 at uhidev0 > uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" > rev 2.00/0.00 addr 2 > uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" > rev 2.00/0.00 addr 2 > vscsi0 at root > scsibus3 at vscsi0: 256 targets > softraid0 at root > scsibus4 at softraid0: 256 targets > root on sd1a (7e3675a2dbcdd669.a) swap on sd1b dump on sd1b > lock order reversal: > 1st 0xffffff041c86cdd8 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4288 > 2nd 0xffffff03a140bd68 inode (&ip->i_lock) @ > /usr/src/sys/ufs/ufs/ufs_vnops.c:1554 > Stopped at db_enter+0x5: popq %rbp > ddb{2}> db_enter(10,ffff800032f93e30,286,8,ffffffff81552a05,0) at db_enter+0x5 > witness_checkorder(ffffff03a140bd58,ffffffff81876bd6,612,9,81,61232f93ed0) at > witness_checkorder+0xaaa > _rw_enter(ffffff03a151a900,ffff8000ffff9410,ffffff039ea5abe0,ffffff03a151a900,ffffff03a140bd58,e578bfe1e390cb1d) > at _rw_enter+0x55 > _rrw_enter(81,ffffff03a140bd58,ffffffff8143e021,ffff800032f93f70,ffffff03a151a900,ffff8000ffff9410) > at _rrw_enter+0x31 > VOP_LOCK(e578bfe1e390cb1d,ffff8000ffff9410,ffffffff00000081,ffffff03a151a900,ffffffff8114a3b1,ffff800032f93fa0) > at VOP_LOCK+0x31 > vn_lock(3000,ffffff039ea5ac04,ffffff039ea5ac00,ffff8000000ce000,ffffffff814a6fd6,ffff800032f93fd0) > at vn_lock+0x36 > uvn_io(ffffff039ea5abe0,ffff800032f94134,ffff800032f94138,ffffff011929ec60,0,e578bfe1e390cb1d) > at uvn_io+0x196 > uvm_pager_put(0,ffffff039ea5abe0,3,3000,0,ffff0100000023) at > uvm_pager_put+0xf6 > uvn_flush(16,3,5b613545000,ffffff041c86cdc0,ffffff039ed6b2c8,e578bfe1e390cb1d) > at uvn_flush+0x414 > uvm_map_clean(ffff800032f94350,1000,ffff8000ffff9410,0,0,0) at > uvm_map_clean+0x3b2 > syscall() at syscall+0x26f > --- syscall (number 256) --- > > syscall 256: msync > > ddb{2}> PID TID PPID UID S FLAGS WAIT > COMMAND > 58977 286365 74422 97 7 0x100010 nsd > 28780 356041 33889 0 7 0x3 unbound-ancho > r > 33889 483347 82627 0 3 0x10008b pause sh > 74422 127608 11923 97 7 0x100010 nsd > *11923 380801 1 97 7 0x100010 nsd > 15691 401174 60684 0 3 0x83 netcon2 nsd-control > 60684 325155 1 0 3 0x100089 pause sh > 99327 483568 25389 74 7 0x100012 pflogd > 25389 397912 1 0 3 0x80 netio pflogd > 82105 37970 79884 0 3 0x100083 nanosleep sleep > 79884 274312 1 0 3 0x100089 pause sh > 47206 259428 63258 73 3 0x100090 kqread syslogd > 63258 318514 1 0 3 0x100082 netio syslogd > 7284 432776 57552 0 3 0x100083 nanosleep sleep > 57552 206272 1 0 3 0x100089 pause sh > 83740 269622 91900 115 3 0x100092 kqread slaacd > 94892 20474 91900 115 3 0x100092 kqread slaacd > 91900 372270 1 0 3 0x80 kqread slaacd > 82627 31822 1 0 3 0x10008b pause sh > 32205 11000 0 0 3 0x14200 pgzero zerothread > 48666 278993 0 0 3 0x14200 aiodoned aiodoned > 98945 425376 0 0 3 0x14200 syncer update > 57380 211885 0 0 3 0x14200 cleaner cleaner > 60523 485933 0 0 3 0x14200 reaper reaper > 59005 88656 0 0 3 0x14200 pgdaemon pagedaemon > 97138 22012 0 0 3 0x14200 bored crynlk > 99275 229256 0 0 3 0x14200 bored crypto > 22598 359088 0 0 3 0x14200 usbtsk usbtask > 78577 73837 0 0 3 0x14200 usbatsk usbatsk > 69923 464026 0 0 3 0x14200 bored i915-hangchec > k > 83143 200995 0 0 3 0x14200 bored i915-dp > 19762 395791 0 0 3 0x14200 bored i915 > 30743 95118 0 0 3 0x40014200 acpi0 acpi0 > 21967 22061 0 0 7 0x40014200 idle7 > 37037 122725 0 0 7 0x40014200 idle6 > 54176 86948 0 0 3 0x40014200 idle5 > 24075 170172 0 0 7 0x40014200 idle4 > 22271 378890 0 0 3 0x40014200 idle3 > 29153 398323 0 0 3 0x40014200 idle2 > 21511 427093 0 0 3 0x40014200 idle1 > 29105 218719 0 0 3 0x14200 bored sensors > 36657 326264 0 0 3 0x14200 bored softnet > 71913 427365 0 0 3 0x14200 bored systqmp > 62318 22393 0 0 3 0x14200 bored systq > 24917 294467 0 0 3 0x40014200 bored softclock > 78233 7063 0 0 3 0x40014200 idle0 > 84793 9847 0 0 3 0x14200 bored sbar > 1 169260 0 0 3 0x82 wait init > 0 0 -1 0 3 0x10200 scheduler swapper > >
