On Fri, 7 May 2021, Greg A. Woods wrote: > I wonder if you could use crash(8) (or ddb(4)) to get kernel stack > traces of the stuck processes. (E.g. see the EXAMPLES section in > crash(8).) > > That might help narrow down which locks are causing the problems...
$ sudo sh -c "echo ps | crash | grep sync | cut -c 30-46 | sed 's,^,bt/a ,' | crash" Crash version 9.1_STABLE, image version 9.1_STABLE. Output from a running system is unreliable. trace: pid 2923 lid 1 at 0xffff810060d26e80 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de mutex_vector_enter() at mutex_vector_enter+0x2b4 do_sys_sync() at do_sys_sync+0x45 sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 75959464276a: trace: pid 5301 lid 1 at 0xffff81007386ee80 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de mutex_vector_enter() at mutex_vector_enter+0x2b4 do_sys_sync() at do_sys_sync+0x45 sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 707b67a4276a: trace: pid 22487 lid 1 at 0xffff810073886e80 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de mutex_vector_enter() at mutex_vector_enter+0x2b4 do_sys_sync() at do_sys_sync+0x45 sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 7a2641a4276a: trace: pid 23870 lid 1 at 0xffff81006038ae80 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de mutex_vector_enter() at mutex_vector_enter+0x2b4 do_sys_sync() at do_sys_sync+0x45 sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 7c73c264276a: trace: pid 21539 lid 1 at 0xffff81005f4fee80 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de mutex_vector_enter() at mutex_vector_enter+0x2b4 do_sys_sync() at do_sys_sync+0x45 sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 72a226c4276a: trace: pid 22210 lid 1 at 0xffff81006078ed20 sleepq_block() at sleepq_block+0x19a turnstile_block() at turnstile_block+0x2de rw_vector_enter() at rw_vector_enter+0x11b genfs_lock() at genfs_lock+0x40 VOP_LOCK() at VOP_LOCK+0x8b vn_lock() at vn_lock+0x11 ffs_sync() at ffs_sync+0x98 VFS_SYNC() at VFS_SYNC+0x35 do_sys_sync() at do_sys_sync+0x6c sys_sync() at sys_sync+0x9 syscall() at syscall+0x157 --- syscall (number 36) --- 78391de4276a: trace: pid 0 lid 67 at 0xffff81005f300f40 sleepq_block() at sleepq_block+0xbb kpause() at kpause+0x10b sched_sync() at sched_sync+0x1e8 -- |/"\ John D. Baker, KN5UKS NetBSD Darwin/MacOS X |\ / jdbaker[snail]consolidated[flyspeck]net OpenBSD FreeBSD | X No HTML/proprietary data in email. BSD just sits there and works! |/ \ GPGkeyID: D703 4A7E 479F 63F8 D3F4 BD99 9572 8F23 E4AD 1645
