Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-12 Thread Bryan Drewery
On 3/5/2014 1:50 PM, Konstantin Belousov wrote:
 On Wed, Mar 05, 2014 at 02:21:04PM -0500, John Baldwin wrote:
 On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote:
 On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote:
 on 04/03/2014 18:45 John Baldwin said the following:
 So I'm not sure how to fix this.  The crash is in this code in 
 vm_object_deallocate():

   if (object-type == OBJT_SWAP 
   (object-flags  OBJ_TMPFS) != 0) {
   vp = object-un_pager.swp.swp_tmpfs;
   vhold(vp);
   VM_OBJECT_WUNLOCK(object);
   vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
   vdrop(vp);
   VM_OBJECT_WLOCK(object);
   if (object-type == OBJT_DEAD ||
   object-ref_count != 1) {
   VM_OBJECT_WUNLOCK(object);
   VOP_UNLOCK(vp, 0);
   return;
   }
   if ((object-flags  OBJ_TMPFS) != 0)
   VOP_UNSET_TEXT(vp);
   VOP_UNLOCK(vp, 0);
   }

 The vdrop() is dropping the count to zero and trying to free the vnode.  
 The 
 real problem I think is that swp_tmpfs doesn't have an implicit vhold() 
 on the 
 vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an 
 already-
 free vnode.  For OBJT_VNODE objects, the reference from the object back 
 to the 
 vnode holds a vref() that gets released by a vput() in 
 vm_object_vndeallocate().

 One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
 untested but might work (but I'm also not sure that this is the right 
 thing in 
 that I don't know what other effects it might have).

 I agree with your analysis, but I don't think that a filesystem holding 
 its own
 vnode is a good idea.  If I am not mistaken, that would prevent tmpfs 
 vnodes
 from going to free list.
 I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could 
 be
 called after VOP_UNLOCK().  Alternatively, the code could handle a doomed 
 vnode
 in a different way.

 I agree with Andrey, it is just a bug to vdrop() before unlock.
 Please try this.

 Ok, my only worry is in the case of Bryan's panic, the hold count on the 
 vnode
 was already zero before vhold() was called, so is it possible that it is a 
 stale
 pointer or is there some other implicit reference that prevents that?  If it 
 can't
 be stale, I think deferring the vdrop() is fine.
 
 The object-un_pager.swp.swp_tmpfs is cleared under the object lock
 before the vnode is reclaimed, i.e. long before the vnode can be freed.
 swp_tmpfs should be kept in sync with the OBJ_TMPFS flag, so the
 vhold() is safe while flag is set and object is locked.
 

This has been stable. Not sure if it fixes the original problem, but I
have not seen any panics since using it and doing many poudriere builds.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-05 Thread Andriy Gapon
on 04/03/2014 18:45 John Baldwin said the following:
 So I'm not sure how to fix this.  The crash is in this code in 
 vm_object_deallocate():
 
   if (object-type == OBJT_SWAP 
   (object-flags  OBJ_TMPFS) != 0) {
   vp = object-un_pager.swp.swp_tmpfs;
   vhold(vp);
   VM_OBJECT_WUNLOCK(object);
   vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
   vdrop(vp);
   VM_OBJECT_WLOCK(object);
   if (object-type == OBJT_DEAD ||
   object-ref_count != 1) {
   VM_OBJECT_WUNLOCK(object);
   VOP_UNLOCK(vp, 0);
   return;
   }
   if ((object-flags  OBJ_TMPFS) != 0)
   VOP_UNSET_TEXT(vp);
   VOP_UNLOCK(vp, 0);
   }
 
 The vdrop() is dropping the count to zero and trying to free the vnode.  The 
 real problem I think is that swp_tmpfs doesn't have an implicit vhold() on 
 the 
 vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already-
 free vnode.  For OBJT_VNODE objects, the reference from the object back to 
 the 
 vnode holds a vref() that gets released by a vput() in 
 vm_object_vndeallocate().
 
 One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
 untested but might work (but I'm also not sure that this is the right thing 
 in 
 that I don't know what other effects it might have).

I agree with your analysis, but I don't think that a filesystem holding its own
vnode is a good idea.  If I am not mistaken, that would prevent tmpfs vnodes
from going to free list.
I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could be
called after VOP_UNLOCK().  Alternatively, the code could handle a doomed vnode
in a different way.

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-05 Thread Konstantin Belousov
On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote:
 on 04/03/2014 18:45 John Baldwin said the following:
  So I'm not sure how to fix this.  The crash is in this code in 
  vm_object_deallocate():
  
  if (object-type == OBJT_SWAP 
  (object-flags  OBJ_TMPFS) != 0) {
  vp = object-un_pager.swp.swp_tmpfs;
  vhold(vp);
  VM_OBJECT_WUNLOCK(object);
  vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
  vdrop(vp);
  VM_OBJECT_WLOCK(object);
  if (object-type == OBJT_DEAD ||
  object-ref_count != 1) {
  VM_OBJECT_WUNLOCK(object);
  VOP_UNLOCK(vp, 0);
  return;
  }
  if ((object-flags  OBJ_TMPFS) != 0)
  VOP_UNSET_TEXT(vp);
  VOP_UNLOCK(vp, 0);
  }
  
  The vdrop() is dropping the count to zero and trying to free the vnode.  
  The 
  real problem I think is that swp_tmpfs doesn't have an implicit vhold() on 
  the 
  vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an 
  already-
  free vnode.  For OBJT_VNODE objects, the reference from the object back to 
  the 
  vnode holds a vref() that gets released by a vput() in 
  vm_object_vndeallocate().
  
  One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
  untested but might work (but I'm also not sure that this is the right thing 
  in 
  that I don't know what other effects it might have).
 
 I agree with your analysis, but I don't think that a filesystem holding its 
 own
 vnode is a good idea.  If I am not mistaken, that would prevent tmpfs vnodes
 from going to free list.
 I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could be
 called after VOP_UNLOCK().  Alternatively, the code could handle a doomed 
 vnode
 in a different way.

I agree with Andrey, it is just a bug to vdrop() before unlock.
Please try this.

diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c
index 8683e2f..787b18b 100644
--- a/sys/vm/vm_object.c
+++ b/sys/vm/vm_object.c
@@ -536,17 +536,18 @@ vm_object_deallocate(vm_object_t object)
vhold(vp);
VM_OBJECT_WUNLOCK(object);
vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
-   vdrop(vp);
VM_OBJECT_WLOCK(object);
if (object-type == OBJT_DEAD ||
object-ref_count != 1) {
VM_OBJECT_WUNLOCK(object);
VOP_UNLOCK(vp, 0);
+   vdrop(vp);
return;
}
if ((object-flags  OBJ_TMPFS) != 0)
VOP_UNSET_TEXT(vp);
VOP_UNLOCK(vp, 0);
+   vdrop(vp);
}
if (object-shadow_count == 0 
object-handle == NULL 


pgpgDGCxvlqyK.pgp
Description: PGP signature


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-05 Thread Bryan Drewery
On 3/5/2014 5:07 AM, Konstantin Belousov wrote:
 On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote:
 on 04/03/2014 18:45 John Baldwin said the following:
 So I'm not sure how to fix this.  The crash is in this code in 
 vm_object_deallocate():

 if (object-type == OBJT_SWAP 
 (object-flags  OBJ_TMPFS) != 0) {
 vp = object-un_pager.swp.swp_tmpfs;
 vhold(vp);
 VM_OBJECT_WUNLOCK(object);
 vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
 vdrop(vp);
 VM_OBJECT_WLOCK(object);
 if (object-type == OBJT_DEAD ||
 object-ref_count != 1) {
 VM_OBJECT_WUNLOCK(object);
 VOP_UNLOCK(vp, 0);
 return;
 }
 if ((object-flags  OBJ_TMPFS) != 0)
 VOP_UNSET_TEXT(vp);
 VOP_UNLOCK(vp, 0);
 }

 The vdrop() is dropping the count to zero and trying to free the vnode.  
 The 
 real problem I think is that swp_tmpfs doesn't have an implicit vhold() on 
 the 
 vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an 
 already-
 free vnode.  For OBJT_VNODE objects, the reference from the object back to 
 the 
 vnode holds a vref() that gets released by a vput() in 
 vm_object_vndeallocate().

 One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
 untested but might work (but I'm also not sure that this is the right thing 
 in 
 that I don't know what other effects it might have).

 I agree with your analysis, but I don't think that a filesystem holding its 
 own
 vnode is a good idea.  If I am not mistaken, that would prevent tmpfs vnodes
 from going to free list.
 I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could be
 called after VOP_UNLOCK().  Alternatively, the code could handle a doomed 
 vnode
 in a different way.
 
 I agree with Andrey, it is just a bug to vdrop() before unlock.
 Please try this.
 
 diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c
 index 8683e2f..787b18b 100644
 --- a/sys/vm/vm_object.c
 +++ b/sys/vm/vm_object.c
 @@ -536,17 +536,18 @@ vm_object_deallocate(vm_object_t object)
   vhold(vp);
   VM_OBJECT_WUNLOCK(object);
   vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
 - vdrop(vp);
   VM_OBJECT_WLOCK(object);
   if (object-type == OBJT_DEAD ||
   object-ref_count != 1) {
   VM_OBJECT_WUNLOCK(object);
   VOP_UNLOCK(vp, 0);
 + vdrop(vp);
   return;
   }
   if ((object-flags  OBJ_TMPFS) != 0)
   VOP_UNSET_TEXT(vp);
   VOP_UNLOCK(vp, 0);
 + vdrop(vp);
   }
   if (object-shadow_count == 0 
   object-handle == NULL 
 

Ok I will try this.

Note that I cannot easily reproduce the issue. So if this seems to be
right logically I suggest just committing after stress testing. I'll
report back after a few builds to let you know if it makes it any worse.

Thanks all.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-05 Thread John Baldwin
On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote:
 On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote:
  on 04/03/2014 18:45 John Baldwin said the following:
   So I'm not sure how to fix this.  The crash is in this code in 
   vm_object_deallocate():
   
 if (object-type == OBJT_SWAP 
 (object-flags  OBJ_TMPFS) != 0) {
 vp = object-un_pager.swp.swp_tmpfs;
 vhold(vp);
 VM_OBJECT_WUNLOCK(object);
 vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
 vdrop(vp);
 VM_OBJECT_WLOCK(object);
 if (object-type == OBJT_DEAD ||
 object-ref_count != 1) {
 VM_OBJECT_WUNLOCK(object);
 VOP_UNLOCK(vp, 0);
 return;
 }
 if ((object-flags  OBJ_TMPFS) != 0)
 VOP_UNSET_TEXT(vp);
 VOP_UNLOCK(vp, 0);
 }
   
   The vdrop() is dropping the count to zero and trying to free the vnode.  
   The 
   real problem I think is that swp_tmpfs doesn't have an implicit vhold() 
   on the 
   vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an 
   already-
   free vnode.  For OBJT_VNODE objects, the reference from the object back 
   to the 
   vnode holds a vref() that gets released by a vput() in 
   vm_object_vndeallocate().
   
   One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
   untested but might work (but I'm also not sure that this is the right 
   thing in 
   that I don't know what other effects it might have).
  
  I agree with your analysis, but I don't think that a filesystem holding its 
  own
  vnode is a good idea.  If I am not mistaken, that would prevent tmpfs vnodes
  from going to free list.
  I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could be
  called after VOP_UNLOCK().  Alternatively, the code could handle a doomed 
  vnode
  in a different way.
 
 I agree with Andrey, it is just a bug to vdrop() before unlock.
 Please try this.

Ok, my only worry is in the case of Bryan's panic, the hold count on the vnode
was already zero before vhold() was called, so is it possible that it is a stale
pointer or is there some other implicit reference that prevents that?  If it 
can't
be stale, I think deferring the vdrop() is fine.

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-05 Thread Konstantin Belousov
On Wed, Mar 05, 2014 at 02:21:04PM -0500, John Baldwin wrote:
 On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote:
  On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote:
   on 04/03/2014 18:45 John Baldwin said the following:
So I'm not sure how to fix this.  The crash is in this code in 
vm_object_deallocate():

if (object-type == OBJT_SWAP 
(object-flags  OBJ_TMPFS) != 0) {
vp = object-un_pager.swp.swp_tmpfs;
vhold(vp);
VM_OBJECT_WUNLOCK(object);
vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
vdrop(vp);
VM_OBJECT_WLOCK(object);
if (object-type == OBJT_DEAD ||
object-ref_count != 1) {
VM_OBJECT_WUNLOCK(object);
VOP_UNLOCK(vp, 0);
return;
}
if ((object-flags  OBJ_TMPFS) != 0)
VOP_UNSET_TEXT(vp);
VOP_UNLOCK(vp, 0);
}

The vdrop() is dropping the count to zero and trying to free the vnode. 
 The 
real problem I think is that swp_tmpfs doesn't have an implicit vhold() 
on the 
vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an 
already-
free vnode.  For OBJT_VNODE objects, the reference from the object back 
to the 
vnode holds a vref() that gets released by a vput() in 
vm_object_vndeallocate().

One fix might be to chagne smp_tmpfs to hold a vhold reference.  This 
is 
untested but might work (but I'm also not sure that this is the right 
thing in 
that I don't know what other effects it might have).
   
   I agree with your analysis, but I don't think that a filesystem holding 
   its own
   vnode is a good idea.  If I am not mistaken, that would prevent tmpfs 
   vnodes
   from going to free list.
   I'd rather try to modify vm_object_deallocate() code.  E.g. vdrop() could 
   be
   called after VOP_UNLOCK().  Alternatively, the code could handle a doomed 
   vnode
   in a different way.
  
  I agree with Andrey, it is just a bug to vdrop() before unlock.
  Please try this.
 
 Ok, my only worry is in the case of Bryan's panic, the hold count on the vnode
 was already zero before vhold() was called, so is it possible that it is a 
 stale
 pointer or is there some other implicit reference that prevents that?  If it 
 can't
 be stale, I think deferring the vdrop() is fine.

The object-un_pager.swp.swp_tmpfs is cleared under the object lock
before the vnode is reclaimed, i.e. long before the vnode can be freed.
swp_tmpfs should be kept in sync with the OBJ_TMPFS flag, so the
vhold() is safe while flag is set and object is locked.


pgpP559xHxzMH.pgp
Description: PGP signature


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-04 Thread John Baldwin
On Monday, March 03, 2014 9:49:39 pm Bryan Drewery wrote:
 On 3/3/2014 12:06 PM, John Baldwin wrote:
  On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote:
  On 2/28/2014 3:18 PM, John Baldwin wrote:
  On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
  While using poudriere:
 
  Unread portion of the kernel message buffer:
  panic: lockmgr still held
  cpuid = 12
  KDB: stack backtrace:
  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfe124804f7a0
  kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850
  vpanic() at vpanic+0x126/frame 0xfe124804f890
  kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900
  lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920
  vdropl() at vdropl+0x1c8/frame 0xfe124804f960
  vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
0xfe124804f9c0
  vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
0xfe124804f9f0
  vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20
  vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60
  exit1() at exit1+0x541/frame 0xfe124804fad0
  sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0
  ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0
  Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0
  --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
0xc45c, rbp = 0xc468 ---
 
  #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
/usr/src/sys/kern/kern_lock.c:440
  440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still 
held));
  (kgdb) print *lk
  $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 
116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 
  18446735288132049184, lk_exslpfail = 0,
lk_timo = 51, lk_pri = 96}
 
  Can you please grab people.freebsd.org/~jhb/gdb/*
 
  and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 
'lockmgr_owner lk'?
 
 
  (kgdb) lockmgr_owner lk
  td: 0xf80272c61920
  pid: 55040, p_comm: testprog
  
  Can you get a stack trace of that program?  ('proc 55040', 'bt')
  
 
 It's the same as original:

So I'm not sure how to fix this.  The crash is in this code in 
vm_object_deallocate():

if (object-type == OBJT_SWAP 
(object-flags  OBJ_TMPFS) != 0) {
vp = object-un_pager.swp.swp_tmpfs;
vhold(vp);
VM_OBJECT_WUNLOCK(object);
vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
vdrop(vp);
VM_OBJECT_WLOCK(object);
if (object-type == OBJT_DEAD ||
object-ref_count != 1) {
VM_OBJECT_WUNLOCK(object);
VOP_UNLOCK(vp, 0);
return;
}
if ((object-flags  OBJ_TMPFS) != 0)
VOP_UNSET_TEXT(vp);
VOP_UNLOCK(vp, 0);
}

The vdrop() is dropping the count to zero and trying to free the vnode.  The 
real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the 
vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already-
free vnode.  For OBJT_VNODE objects, the reference from the object back to the 
vnode holds a vref() that gets released by a vput() in 
vm_object_vndeallocate().

One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
untested but might work (but I'm also not sure that this is the right thing in 
that I don't know what other effects it might have).  Can you reproduce this 
easily?

Index: fs/tmpfs/tmpfs_subr.c
===
--- fs/tmpfs/tmpfs_subr.c   (revision 262711)
+++ fs/tmpfs/tmpfs_subr.c   (working copy)
@@ -440,10 +440,11 @@
 
VM_OBJECT_WLOCK(obj);
VI_LOCK(vp);
+   KASSERT(obj-un_pager.swp.swp_tmpfs == vp, vp mismatch);
vm_object_clear_flag(obj, OBJ_TMPFS);
obj-un_pager.swp.swp_tmpfs = NULL;
-   VI_UNLOCK(vp);
VM_OBJECT_WUNLOCK(obj);
+   vdropl(vp);
 }
 
 /*
@@ -578,6 +579,7 @@
VI_LOCK(vp);
KASSERT(vp-v_object == NULL, (Not NULL v_object in tmpfs));
vp-v_object = object;
+   vholdl(vp);
object-un_pager.swp.swp_tmpfs = vp;
vm_object_set_flag(object, OBJ_TMPFS);
VI_UNLOCK(vp);
Index: vm/vm_object.c
===
--- vm/vm_object.c  (revision 262711)
+++ vm/vm_object.c  (working copy)
@@ -547,6 +547,7 @@
if ((object-flags  OBJ_TMPFS) != 0)
 

Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-03 Thread John Baldwin
On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote:
 On 2/28/2014 3:18 PM, John Baldwin wrote:
  On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
  While using poudriere:
 
  Unread portion of the kernel message buffer:
  panic: lockmgr still held
  cpuid = 12
  KDB: stack backtrace:
  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
  0xfe124804f7a0
  kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850
  vpanic() at vpanic+0x126/frame 0xfe124804f890
  kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900
  lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920
  vdropl() at vdropl+0x1c8/frame 0xfe124804f960
  vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
  0xfe124804f9c0
  vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
  0xfe124804f9f0
  vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20
  vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60
  exit1() at exit1+0x541/frame 0xfe124804fad0
  sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0
  ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0
  Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0
  --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
  0xc45c, rbp = 0xc468 ---
 
  #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
  /usr/src/sys/kern/kern_lock.c:440
  440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still 
  held));
  (kgdb) print *lk
  $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 
  116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 
  18446735288132049184, lk_exslpfail = 0,
lk_timo = 51, lk_pri = 96}
  
  Can you please grab people.freebsd.org/~jhb/gdb/*
  
  and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner 
  lk'?
  
 
 (kgdb) lockmgr_owner lk
 td: 0xf80272c61920
 pid: 55040, p_comm: testprog

Can you get a stack trace of that program?  ('proc 55040', 'bt')

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-03 Thread Bryan Drewery
On 3/3/2014 12:06 PM, John Baldwin wrote:
 On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote:
 On 2/28/2014 3:18 PM, John Baldwin wrote:
 On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
 While using poudriere:

 Unread portion of the kernel message buffer:
 panic: lockmgr still held
 cpuid = 12
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
 0xfe124804f7a0
 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850
 vpanic() at vpanic+0x126/frame 0xfe124804f890
 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900
 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920
 vdropl() at vdropl+0x1c8/frame 0xfe124804f960
 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
 0xfe124804f9c0
 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
 0xfe124804f9f0
 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20
 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60
 exit1() at exit1+0x541/frame 0xfe124804fad0
 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0
 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0
 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0
 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
 0xc45c, rbp = 0xc468 ---

 #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
 /usr/src/sys/kern/kern_lock.c:440
 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still 
 held));
 (kgdb) print *lk
 $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 
 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 
 18446735288132049184, lk_exslpfail = 0,
   lk_timo = 51, lk_pri = 96}

 Can you please grab people.freebsd.org/~jhb/gdb/*

 and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner 
 lk'?


 (kgdb) lockmgr_owner lk
 td: 0xf80272c61920
 pid: 55040, p_comm: testprog
 
 Can you get a stack trace of that program?  ('proc 55040', 'bt')
 

It's the same as original:

 Loaded symbols for /boot/kernel/sem.ko.symbols
 #0  doadump (textdump=1) at pcpu.h:219
 219 __asm(movq %%gs:%1,%0 : =r (td)
 (kgdb) bt
 #0  doadump (textdump=1) at pcpu.h:219
 #1  0x808e33a7 in kern_reboot (howto=260) at 
 /usr/src/sys/kern/kern_shutdown.c:452
 #2  0x808e38b5 in vpanic (fmt=value optimized out, ap=value 
 optimized out) at /usr/src/sys/kern/kern_shutdown.c:759
 #3  0x808e3749 in kassert_panic (fmt=value optimized out) at 
 /usr/src/sys/kern/kern_shutdown.c:647
 #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
 /usr/src/sys/kern/kern_lock.c:440
 #5  0x80990d38 in vdropl (vp=0xf80a88a28588) at 
 /usr/src/sys/kern/vfs_subr.c:2417
 #6  0x80b73cbb in vm_object_deallocate (object=value optimized out) 
 at /usr/src/sys/vm/vm_object.c:539
 #7  0x80b6a3a9 in vm_map_process_deferred () at 
 /usr/src/sys/vm/vm_map.c:2763
 #8  0x80b6e9c8 in vm_map_remove (map=value optimized out, 
 start=value optimized out, end=value optimized out) at 
 /usr/src/sys/vm/vm_map.c:488
 #9  0x80b6a179 in vmspace_exit (td=0xf80272c61920) at 
 /usr/src/sys/vm/vm_map.c:327
 #10 0x808a7411 in exit1 (td=0xf80272c61920, rv=value optimized 
 out) at /usr/src/sys/kern/kern_exit.c:319
 #11 0x808a6ece in sys_sys_exit (td=0x0, uap=value optimized out) at 
 /usr/src/sys/kern/kern_exit.c:119
 #12 0x80dd2720 in ia32_syscall (frame=0xfe124804fc00) at 
 subr_syscall.c:133
 #13 0x80cd6325 in Xint0x80_syscall () at ia32_exception.S:73
 #14 0x281014df in ?? ()
 Previous frame inner to this frame (corrupt stack?)
 Current language:  auto; currently minimal
 (kgdb) proc 55040
 [Switching to thread 668 (Thread 100898)]#0  doadump (textdump=1) at 
 pcpu.h:219
 219 __asm(movq %%gs:%1,%0 : =r (td)
 (kgdb) bt
 #0  doadump (textdump=1) at pcpu.h:219
 #1  0x808e33a7 in kern_reboot (howto=260) at 
 /usr/src/sys/kern/kern_shutdown.c:452
 #2  0x808e38b5 in vpanic (fmt=value optimized out, ap=value 
 optimized out) at /usr/src/sys/kern/kern_shutdown.c:759
 #3  0x808e3749 in kassert_panic (fmt=value optimized out) at 
 /usr/src/sys/kern/kern_shutdown.c:647
 #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
 /usr/src/sys/kern/kern_lock.c:440
 #5  0x80990d38 in vdropl (vp=0xf80a88a28588) at 
 /usr/src/sys/kern/vfs_subr.c:2417
 #6  0x80b73cbb in vm_object_deallocate (object=value optimized out) 
 at /usr/src/sys/vm/vm_object.c:539
 #7  0x80b6a3a9 in vm_map_process_deferred () at 
 /usr/src/sys/vm/vm_map.c:2763
 #8  0x80b6e9c8 in vm_map_remove (map=value optimized out, 
 start=value optimized out, end=value optimized out) at 
 /usr/src/sys/vm/vm_map.c:488
 #9  0x80b6a179 in vmspace_exit (td=0xf80272c61920) at 
 /usr/src/sys/vm/vm_map.c:327
 #10 0x808a7411 in exit1 

Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-03-02 Thread Bryan Drewery
On 2/28/2014 3:18 PM, John Baldwin wrote:
 On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
 While using poudriere:

 Unread portion of the kernel message buffer:
 panic: lockmgr still held
 cpuid = 12
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
 0xfe124804f7a0
 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850
 vpanic() at vpanic+0x126/frame 0xfe124804f890
 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900
 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920
 vdropl() at vdropl+0x1c8/frame 0xfe124804f960
 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
 0xfe124804f9c0
 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
 0xfe124804f9f0
 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20
 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60
 exit1() at exit1+0x541/frame 0xfe124804fad0
 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0
 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0
 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0
 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
 0xc45c, rbp = 0xc468 ---

 #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
 /usr/src/sys/kern/kern_lock.c:440
 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held));
 (kgdb) print *lk
 $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 
 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 
 18446735288132049184, lk_exslpfail = 0,
   lk_timo = 51, lk_pri = 96}
 
 Can you please grab people.freebsd.org/~jhb/gdb/*
 
 and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'?
 

(kgdb) lockmgr_owner lk
td: 0xf80272c61920
pid: 55040, p_comm: testprog


-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature


Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)

2014-02-28 Thread John Baldwin
On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
 While using poudriere:
 
  Unread portion of the kernel message buffer:
  panic: lockmgr still held
  cpuid = 12
  KDB: stack backtrace:
  db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
  0xfe124804f7a0
  kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850
  vpanic() at vpanic+0x126/frame 0xfe124804f890
  kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900
  lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920
  vdropl() at vdropl+0x1c8/frame 0xfe124804f960
  vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
  0xfe124804f9c0
  vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
  0xfe124804f9f0
  vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20
  vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60
  exit1() at exit1+0x541/frame 0xfe124804fad0
  sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0
  ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0
  Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0
  --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
  0xc45c, rbp = 0xc468 ---
 
  #4  0x808c00db in lockdestroy (lk=0xf80a88a285f0) at 
  /usr/src/sys/kern/kern_lock.c:440
  440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held));
  (kgdb) print *lk
  $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 
  116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 
18446735288132049184, lk_exslpfail = 0,
lk_timo = 51, lk_pri = 96}

Can you please grab people.freebsd.org/~jhb/gdb/*

and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'?

-- 
John Baldwin
___
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org