Re: zfs/vfs lockups, via poudriere

2014-11-23 Thread Sean Bruno
On Sun, 2014-11-23 at 00:09 +0200, Andriy Gapon wrote:
 On 22/11/2014 21:20, Sean Bruno wrote:
  bdrewery reported a vfs/zfs condition where operations will stall out
  and block (rm, mv, file) during a poudriere build.  I've hit this now
  and it seems to be alleviated by setting vfs.lookup_shared=0
  
  I seem to be able to trivially reproduce this on my builders and want to
  know if anyone is looking to diagnose this further.
  
  original message:
  https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html
  
  On my builders I see:
  
  procstat -kka | grep zfs
  
  0 100666 kernel   zfs_vn_rele_task mi_switch+0xe1 
  sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a 
  fork_trampoline+0xe 
  3 100151 zfskern  arc_reclaim_thre mi_switch+0xe1 
  sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 
  fork_exit+0x9a fork_trampoline+0xe 
  3 100152 zfskern  l2arc_feed_threa mi_switch+0xe1 
  sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f 
  fork_exit+0x9a fork_trampoline+0xe 
  3 100657 zfskern  trim zroot   mi_switch+0xe1 
  sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e 
  fork_exit+0x9a fork_trampoline+0xe 
  3 100675 zfskern  txg_thread_enter mi_switch+0xe1 
  sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a 
  fork_trampoline+0xe 
  3 100676 zfskern  txg_thread_enter mi_switch+0xe1 
  sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc 
  fork_exit+0x9a fork_trampoline+0xe 
  31071 100995 rm   -mi_switch+0xe1 
  sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c 
  VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d 
  VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 
  lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb 
  Xfast_syscall+0xfb 
  31075 100693 mv   -mi_switch+0xe1 
  sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c 
  VOP_LOCK1_APV+0xab _vn_lock+0x4
 
 The last line looks incomplete.
 
 
hrm ... cut-n-paste fail I guess.

procstat -kka | grep zfs

0 100666 kernel   zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a 
_sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe 
3 100151 zfskern  arc_reclaim_thre mi_switch+0xe1 
sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 
fork_exit+0x9a fork_trampoline+0xe 
3 100152 zfskern  l2arc_feed_threa mi_switch+0xe1 
sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f 
fork_exit+0x9a fork_trampoline+0xe 
3 100657 zfskern  trim zroot   mi_switch+0xe1 
sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a 
fork_trampoline+0xe 
3 100675 zfskern  txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a 
_cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe 
3 100676 zfskern  txg_thread_enter mi_switch+0xe1 
sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc 
fork_exit+0x9a fork_trampoline+0xe 
31071 100995 rm   -mi_switch+0xe1 sleepq_wait+0x3a 
sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab 
_vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d 
VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 
lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb 
Xfast_syscall+0xfb 
31075 100693 mv   -mi_switch+0xe1 sleepq_wait+0x3a 
sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab 
_vn_lock+0x43 vputx+0x28a zfs_rename_unlock+0x3e zfs_freebsd_rename+0xe39 
VOP_RENAME_APV+0xab kern_renameat+0x4a6 amd64_syscall+0x3fb Xfast_syscall+0xfb 





signature.asc
Description: This is a digitally signed message part


Re: zfs/vfs lockups, via poudriere

2014-11-22 Thread Andriy Gapon
On 22/11/2014 21:20, Sean Bruno wrote:
 bdrewery reported a vfs/zfs condition where operations will stall out
 and block (rm, mv, file) during a poudriere build.  I've hit this now
 and it seems to be alleviated by setting vfs.lookup_shared=0
 
 I seem to be able to trivially reproduce this on my builders and want to
 know if anyone is looking to diagnose this further.
 
 original message:
 https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html
 
 On my builders I see:
 
 procstat -kka | grep zfs
 
 0 100666 kernel   zfs_vn_rele_task mi_switch+0xe1 
 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a 
 fork_trampoline+0xe 
 3 100151 zfskern  arc_reclaim_thre mi_switch+0xe1 
 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 
 fork_exit+0x9a fork_trampoline+0xe 
 3 100152 zfskern  l2arc_feed_threa mi_switch+0xe1 
 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f 
 fork_exit+0x9a fork_trampoline+0xe 
 3 100657 zfskern  trim zroot   mi_switch+0xe1 
 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a 
 fork_trampoline+0xe 
 3 100675 zfskern  txg_thread_enter mi_switch+0xe1 
 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a 
 fork_trampoline+0xe 
 3 100676 zfskern  txg_thread_enter mi_switch+0xe1 
 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc 
 fork_exit+0x9a fork_trampoline+0xe 
 31071 100995 rm   -mi_switch+0xe1 
 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c 
 VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d 
 VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 
 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb 
 Xfast_syscall+0xfb 
 31075 100693 mv   -mi_switch+0xe1 
 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c 
 VOP_LOCK1_APV+0xab _vn_lock+0x4

The last line looks incomplete.


-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org