Just adding more debug info.
lock order reversal:
1st 0xffffff041c86cdd8 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4288
2nd 0xffffff03a2684e70 inode (&ip->i_lock) @
/usr/src/sys/ufs/ufs/ufs_vnops.c:1554
Stopped at db_enter+0x5: popq %rbp
ddb{3}> Number of known direct relationships is 291
Lock order reversal between "&map->lock"(rwlock) and "&ip->i_lock"(rrwlock)!
Lock order "&map->lock"(rwlock) -> "&ip->i_lock"(rrwlock) first seen at:
#0 witness_checkorder+0x463
#1 _rw_enter+0x55
#2 _rrw_enter+0x31
#3 VOP_LOCK+0x31
#4 vn_lock+0x36
#5 uvn_io+0x196
#6 uvm_pager_put+0xf6
#7 uvn_flush+0x414
#8 uvm_map_clean+0x3b2
#9 syscall+0x26f
Lock order "&ip->i_lock"(rrwlock) -> "&map->lock"(rwlock) first seen at:
#0 witness_checkorder+0x463
#1 _rw_enter_read+0x46
#2 uvmfault_lookup+0x7d
#3 uvm_fault+0x64
#4 trap+0x49c
#5 copyout+0x48
#6 ffs_read+0x1d2
#7 VOP_READ+0x39
#8 vn_read+0xcf
#9 dofilereadv+0x244
#10 sys_read+0x76
#11 syscall+0x26f
Lock order reversal between "&mp->mnt_lock"(rwlock) and "&ip->i_lock"(rrwlock)!
Lock order "&mp->mnt_lock"(rwlock) -> "&ip->i_lock"(rrwlock) first seen at:
#0 witness_checkorder+0x463
#1 _rw_enter+0x55
#2 _rrw_enter+0x31
#3 VOP_LOCK+0x31
#4 vn_lock+0x36
#5 ffs_reload+0x59
#6 ffs_mount+0x504
#7 sys_mount+0x2db
#8 syscall+0x26f
Lock order "&ip->i_lock"(rrwlock) -> "&mp->mnt_lock"(rwlock) first seen at:
#0 witness_checkorder+0x463
#1 _rw_enter+0x55
#2 vfs_busy+0x5d
#3 vfs_lookup+0x399
#4 namei+0x1ec
#5 doreadlinkat+0x6d
#6 syscall+0x26f
ddb{3}> shared rwlock vmmaplk r = 0 (0xffffff041c86cdd8) locked @
/usr/src/sys/uvm/uvm_map.c:4288
exclusive kernel_lock &kernel_lock r = 0 (0xffffffff81bc8570) locked @
/usr/src/sys/sys/syscall_mi.h:69
On Fri, Nov 10, 2017 at 07:25:24AM -0500, David Hill wrote:
> Hello -
>
> I enabled witness(4) and vfs lock in hopes to discover a hardlock after
> Oct 29. I drop into the debugger on boot when nsd(8) starts:
>
> OpenBSD 6.2-current (GENERIC.MP) #37: Thu Nov 9 09:52:26 EST 2017
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17032458240 (16243MB)
> avail mem = 16378048512 (15619MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb450 (75 entries)
> bios0: vendor American Megatrends Inc. version "F15" date 10/23/2013
> bios0: Gigabyte Technology Co., Ltd. B75M-D3H
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP APIC MCFG HPET SSDT SSDT SSDT DMAR
> acpi0: wakeup devices PS2K(S3) P0P1(S4) USB1(S3) USB2(S3) USB3(S3) USB4(S3)
> USB5(S3) USB6(S3) USB7(S3) PXSX(S4) RP01(
> S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.78 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
> E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu0: 256KB 64b/line 8-way L2 cache
> acpitimer0: recalibrated TSC frequency 3392291239 Hz
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
> E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 1 (application processor)
> cpu4: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 1, core 0, package 0
> cpu5 at mainbus0: apid 3 (application processor)
> cpu5: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 1, package 0
> cpu6 at mainbus0: apid 5 (application processor)
> cpu6: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu6:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 1, core 2, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu7:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
> acpihpet0 at acpi0: 14318179 Hz
> acpihpet0: recalibrated TSC frequency 3392306800 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 5 (P0P1)
> acpiprt2 at acpi0: bus 1 (RP01)
> acpiprt3 at acpi0: bus -1 (RP02)
> acpiprt4 at acpi0: bus -1 (RP03)
> acpiprt5 at acpi0: bus -1 (RP04)
> acpiprt6 at acpi0: bus 4 (RP05)
> acpiprt7 at acpi0: bus -1 (RP06)
> acpiprt8 at acpi0: bus -1 (RP07)
> acpiprt9 at acpi0: bus -1 (RP08)
> acpiprt10 at acpi0: bus -1 (PEG0)
> acpiprt11 at acpi0: bus -1 (PEG1)
> acpiprt12 at acpi0: bus -1 (PEG2)
> acpiprt13 at acpi0: bus -1 (PEG3)
> acpiec0 at acpi0: not present
> acpicpu0 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu4 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu5 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu6 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu7 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: FN00, resource for FAN0
> acpipwrres1 at acpi0: FN01, resource for FAN1
> acpipwrres2 at acpi0: FN02, resource for FAN2
> acpipwrres3 at acpi0: FN03, resource for FAN3
> acpipwrres4 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 106 degC
> "INT3F0D" at acpi0 not configured
> "PNP0400" at acpi0 not configured
> acpibtn0 at acpi0: PWRB
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD02
> cpu0: Enhanced SpeedStep 3392 MHz: speeds: 3901, 3900, 3700, 3600, 3400,
> 3200, 3100, 2900, 2700, 2600, 2400, 2300, 2100, 1900, 1800, 1600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09
> inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09
> drm0 at inteldrm0
> inteldrm0: msi
> inteldrm0: 1280x1024, 32bpp
> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> xhci0 at pci0 dev 20 function 0 "Intel 7 Series xHCI" rev 0x04: msi
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
> addr 1
> "Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 16
> usb1 at ehci0: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
> addr 1
> ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi
> pci1 at ppb0 bus 1
> ppb1 at pci1 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci2 at ppb1 bus 2
> mfi0 at pci2 dev 14 function 0 "Symbios Logic MegaRAID SAS 1064R" rev 0x00:
> apic 2 int 18
> mfi0: "MegaRAID SAS 84016E", firmware 7.0.1-0084, 256MB cache
> scsibus1 at mfi0: 64 targets
> sd0 at scsibus1 targ 0 lun 0: <LSI, MegaRAID 84016E, 1.12> SCSI3 0/direct
> fixed naa.600605b001d5a4101c162d40411f21f2
> sd0: 2858160MB, 512 bytes/sector, 5853511680 sectors
> ppb2 at pci1 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci3 at ppb2 bus 3
> ppb3 at pci0 dev 28 function 4 "Intel 7 Series PCIE" rev 0xc4: msi
> pci4 at ppb3 bus 4
> re0 at pci4 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL
> (0x2c80), msi, address fc:aa:14:29:1d:1d
> rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 5
> ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 23
> usb2 at ehci1: USB revision 2.0
> uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
> addr 1
> ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
> pci5 at ppb4 bus 5
> fxp0 at pci5 dev 1 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 19,
> address 00:0e:0c:35:70:48
> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
> pcib0 at pci0 dev 31 function 0 "Intel B75 LPC" rev 0x04
> ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3
> ahci0: port 0: 6.0Gb/s
> scsibus2 at ahci0: 32 targets
> sd1 at scsibus2 targ 0 lun 0: <ATA, Crucial_CT128MX1, MU01> SCSI3 0/direct
> fixed naa.500a07510d04dca2
> sd1: 122104MB, 512 bytes/sector, 250069680 sectors, thin
> ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int
> 18
> iic0 at ichiic0
> spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800
> spdmem1 at iic0 addr 0x53: 8GB DDR3 SDRAM PC3-12800
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> it0 at isa0 port 0x2e/2: IT8728F rev 1, EC port 0xa30
> error: [drm:pid0:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo
> underrun on pch transcoder A
> error: [drm:pid0:intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder
> A FIFO underrun
> uhidev0 at uhub0 port 2 configuration 1 interface 0 "American Power
> Conversion Back-UPS RS 700G FW:856.L5 -P.D USB FW:L5 -P" rev 2.00/0.90 addr 2
> uhidev0: iclass 3/0, 146 report ids
> upd0 at uhidev0
> uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub"
> rev 2.00/0.00 addr 2
> uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub"
> rev 2.00/0.00 addr 2
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd1a (7e3675a2dbcdd669.a) swap on sd1b dump on sd1b
> lock order reversal:
> 1st 0xffffff041c86cdd8 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4288
> 2nd 0xffffff03a140bd68 inode (&ip->i_lock) @
> /usr/src/sys/ufs/ufs/ufs_vnops.c:1554
> Stopped at db_enter+0x5: popq %rbp
> ddb{2}> db_enter(10,ffff800032f93e30,286,8,ffffffff81552a05,0) at db_enter+0x5
> witness_checkorder(ffffff03a140bd58,ffffffff81876bd6,612,9,81,61232f93ed0) at
> witness_checkorder+0xaaa
> _rw_enter(ffffff03a151a900,ffff8000ffff9410,ffffff039ea5abe0,ffffff03a151a900,ffffff03a140bd58,e578bfe1e390cb1d)
> at _rw_enter+0x55
> _rrw_enter(81,ffffff03a140bd58,ffffffff8143e021,ffff800032f93f70,ffffff03a151a900,ffff8000ffff9410)
> at _rrw_enter+0x31
> VOP_LOCK(e578bfe1e390cb1d,ffff8000ffff9410,ffffffff00000081,ffffff03a151a900,ffffffff8114a3b1,ffff800032f93fa0)
> at VOP_LOCK+0x31
> vn_lock(3000,ffffff039ea5ac04,ffffff039ea5ac00,ffff8000000ce000,ffffffff814a6fd6,ffff800032f93fd0)
> at vn_lock+0x36
> uvn_io(ffffff039ea5abe0,ffff800032f94134,ffff800032f94138,ffffff011929ec60,0,e578bfe1e390cb1d)
> at uvn_io+0x196
> uvm_pager_put(0,ffffff039ea5abe0,3,3000,0,ffff0100000023) at
> uvm_pager_put+0xf6
> uvn_flush(16,3,5b613545000,ffffff041c86cdc0,ffffff039ed6b2c8,e578bfe1e390cb1d)
> at uvn_flush+0x414
> uvm_map_clean(ffff800032f94350,1000,ffff8000ffff9410,0,0,0) at
> uvm_map_clean+0x3b2
> syscall() at syscall+0x26f
> --- syscall (number 256) ---
>
> syscall 256: msync
>
> ddb{2}> PID TID PPID UID S FLAGS WAIT
> COMMAND
> 58977 286365 74422 97 7 0x100010 nsd
> 28780 356041 33889 0 7 0x3 unbound-ancho
> r
> 33889 483347 82627 0 3 0x10008b pause sh
> 74422 127608 11923 97 7 0x100010 nsd
> *11923 380801 1 97 7 0x100010 nsd
> 15691 401174 60684 0 3 0x83 netcon2 nsd-control
> 60684 325155 1 0 3 0x100089 pause sh
> 99327 483568 25389 74 7 0x100012 pflogd
> 25389 397912 1 0 3 0x80 netio pflogd
> 82105 37970 79884 0 3 0x100083 nanosleep sleep
> 79884 274312 1 0 3 0x100089 pause sh
> 47206 259428 63258 73 3 0x100090 kqread syslogd
> 63258 318514 1 0 3 0x100082 netio syslogd
> 7284 432776 57552 0 3 0x100083 nanosleep sleep
> 57552 206272 1 0 3 0x100089 pause sh
> 83740 269622 91900 115 3 0x100092 kqread slaacd
> 94892 20474 91900 115 3 0x100092 kqread slaacd
> 91900 372270 1 0 3 0x80 kqread slaacd
> 82627 31822 1 0 3 0x10008b pause sh
> 32205 11000 0 0 3 0x14200 pgzero zerothread
> 48666 278993 0 0 3 0x14200 aiodoned aiodoned
> 98945 425376 0 0 3 0x14200 syncer update
> 57380 211885 0 0 3 0x14200 cleaner cleaner
> 60523 485933 0 0 3 0x14200 reaper reaper
> 59005 88656 0 0 3 0x14200 pgdaemon pagedaemon
> 97138 22012 0 0 3 0x14200 bored crynlk
> 99275 229256 0 0 3 0x14200 bored crypto
> 22598 359088 0 0 3 0x14200 usbtsk usbtask
> 78577 73837 0 0 3 0x14200 usbatsk usbatsk
> 69923 464026 0 0 3 0x14200 bored i915-hangchec
> k
> 83143 200995 0 0 3 0x14200 bored i915-dp
> 19762 395791 0 0 3 0x14200 bored i915
> 30743 95118 0 0 3 0x40014200 acpi0 acpi0
> 21967 22061 0 0 7 0x40014200 idle7
> 37037 122725 0 0 7 0x40014200 idle6
> 54176 86948 0 0 3 0x40014200 idle5
> 24075 170172 0 0 7 0x40014200 idle4
> 22271 378890 0 0 3 0x40014200 idle3
> 29153 398323 0 0 3 0x40014200 idle2
> 21511 427093 0 0 3 0x40014200 idle1
> 29105 218719 0 0 3 0x14200 bored sensors
> 36657 326264 0 0 3 0x14200 bored softnet
> 71913 427365 0 0 3 0x14200 bored systqmp
> 62318 22393 0 0 3 0x14200 bored systq
> 24917 294467 0 0 3 0x40014200 bored softclock
> 78233 7063 0 0 3 0x40014200 idle0
> 84793 9847 0 0 3 0x14200 bored sbar
> 1 169260 0 0 3 0x82 wait init
> 0 0 -1 0 3 0x10200 scheduler swapper
>
>
