Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On 3/5/2014 1:50 PM, Konstantin Belousov wrote: On Wed, Mar 05, 2014 at 02:21:04PM -0500, John Baldwin wrote: On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote: On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote: on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. I agree with Andrey, it is just a bug to vdrop() before unlock. Please try this. Ok, my only worry is in the case of Bryan's panic, the hold count on the vnode was already zero before vhold() was called, so is it possible that it is a stale pointer or is there some other implicit reference that prevents that? If it can't be stale, I think deferring the vdrop() is fine. The object-un_pager.swp.swp_tmpfs is cleared under the object lock before the vnode is reclaimed, i.e. long before the vnode can be freed. swp_tmpfs should be kept in sync with the OBJ_TMPFS flag, so the vhold() is safe while flag is set and object is locked. This has been stable. Not sure if it fixes the original problem, but I have not seen any panics since using it and doing many poudriere builds. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. -- Andriy Gapon ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote: on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. I agree with Andrey, it is just a bug to vdrop() before unlock. Please try this. diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c index 8683e2f..787b18b 100644 --- a/sys/vm/vm_object.c +++ b/sys/vm/vm_object.c @@ -536,17 +536,18 @@ vm_object_deallocate(vm_object_t object) vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); + vdrop(vp); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); + vdrop(vp); } if (object-shadow_count == 0 object-handle == NULL pgpgDGCxvlqyK.pgp Description: PGP signature
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On 3/5/2014 5:07 AM, Konstantin Belousov wrote: On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote: on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. I agree with Andrey, it is just a bug to vdrop() before unlock. Please try this. diff --git a/sys/vm/vm_object.c b/sys/vm/vm_object.c index 8683e2f..787b18b 100644 --- a/sys/vm/vm_object.c +++ b/sys/vm/vm_object.c @@ -536,17 +536,18 @@ vm_object_deallocate(vm_object_t object) vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); - vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); + vdrop(vp); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); + vdrop(vp); } if (object-shadow_count == 0 object-handle == NULL Ok I will try this. Note that I cannot easily reproduce the issue. So if this seems to be right logically I suggest just committing after stress testing. I'll report back after a few builds to let you know if it makes it any worse. Thanks all. -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote: On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote: on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. I agree with Andrey, it is just a bug to vdrop() before unlock. Please try this. Ok, my only worry is in the case of Bryan's panic, the hold count on the vnode was already zero before vhold() was called, so is it possible that it is a stale pointer or is there some other implicit reference that prevents that? If it can't be stale, I think deferring the vdrop() is fine. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Wed, Mar 05, 2014 at 02:21:04PM -0500, John Baldwin wrote: On Wednesday, March 05, 2014 6:07:23 am Konstantin Belousov wrote: On Wed, Mar 05, 2014 at 11:41:24AM +0200, Andriy Gapon wrote: on 04/03/2014 18:45 John Baldwin said the following: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). I agree with your analysis, but I don't think that a filesystem holding its own vnode is a good idea. If I am not mistaken, that would prevent tmpfs vnodes from going to free list. I'd rather try to modify vm_object_deallocate() code. E.g. vdrop() could be called after VOP_UNLOCK(). Alternatively, the code could handle a doomed vnode in a different way. I agree with Andrey, it is just a bug to vdrop() before unlock. Please try this. Ok, my only worry is in the case of Bryan's panic, the hold count on the vnode was already zero before vhold() was called, so is it possible that it is a stale pointer or is there some other implicit reference that prevents that? If it can't be stale, I think deferring the vdrop() is fine. The object-un_pager.swp.swp_tmpfs is cleared under the object lock before the vnode is reclaimed, i.e. long before the vnode can be freed. swp_tmpfs should be kept in sync with the OBJ_TMPFS flag, so the vhold() is safe while flag is set and object is locked. pgpP559xHxzMH.pgp Description: PGP signature
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Monday, March 03, 2014 9:49:39 pm Bryan Drewery wrote: On 3/3/2014 12:06 PM, John Baldwin wrote: On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote: On 2/28/2014 3:18 PM, John Baldwin wrote: On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote: While using poudriere: Unread portion of the kernel message buffer: panic: lockmgr still held cpuid = 12 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe124804f7a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850 vpanic() at vpanic+0x126/frame 0xfe124804f890 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920 vdropl() at vdropl+0x1c8/frame 0xfe124804f960 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 0xfe124804f9c0 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfe124804f9f0 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60 exit1() at exit1+0x541/frame 0xfe124804fad0 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 0xc45c, rbp = 0xc468 --- #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held)); (kgdb) print *lk $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 18446735288132049184, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96} Can you please grab people.freebsd.org/~jhb/gdb/* and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'? (kgdb) lockmgr_owner lk td: 0xf80272c61920 pid: 55040, p_comm: testprog Can you get a stack trace of that program? ('proc 55040', 'bt') It's the same as original: So I'm not sure how to fix this. The crash is in this code in vm_object_deallocate(): if (object-type == OBJT_SWAP (object-flags OBJ_TMPFS) != 0) { vp = object-un_pager.swp.swp_tmpfs; vhold(vp); VM_OBJECT_WUNLOCK(object); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vdrop(vp); VM_OBJECT_WLOCK(object); if (object-type == OBJT_DEAD || object-ref_count != 1) { VM_OBJECT_WUNLOCK(object); VOP_UNLOCK(vp, 0); return; } if ((object-flags OBJ_TMPFS) != 0) VOP_UNSET_TEXT(vp); VOP_UNLOCK(vp, 0); } The vdrop() is dropping the count to zero and trying to free the vnode. The real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already- free vnode. For OBJT_VNODE objects, the reference from the object back to the vnode holds a vref() that gets released by a vput() in vm_object_vndeallocate(). One fix might be to chagne smp_tmpfs to hold a vhold reference. This is untested but might work (but I'm also not sure that this is the right thing in that I don't know what other effects it might have). Can you reproduce this easily? Index: fs/tmpfs/tmpfs_subr.c === --- fs/tmpfs/tmpfs_subr.c (revision 262711) +++ fs/tmpfs/tmpfs_subr.c (working copy) @@ -440,10 +440,11 @@ VM_OBJECT_WLOCK(obj); VI_LOCK(vp); + KASSERT(obj-un_pager.swp.swp_tmpfs == vp, vp mismatch); vm_object_clear_flag(obj, OBJ_TMPFS); obj-un_pager.swp.swp_tmpfs = NULL; - VI_UNLOCK(vp); VM_OBJECT_WUNLOCK(obj); + vdropl(vp); } /* @@ -578,6 +579,7 @@ VI_LOCK(vp); KASSERT(vp-v_object == NULL, (Not NULL v_object in tmpfs)); vp-v_object = object; + vholdl(vp); object-un_pager.swp.swp_tmpfs = vp; vm_object_set_flag(object, OBJ_TMPFS); VI_UNLOCK(vp); Index: vm/vm_object.c === --- vm/vm_object.c (revision 262711) +++ vm/vm_object.c (working copy) @@ -547,6 +547,7 @@ if ((object-flags OBJ_TMPFS) != 0)
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote: On 2/28/2014 3:18 PM, John Baldwin wrote: On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote: While using poudriere: Unread portion of the kernel message buffer: panic: lockmgr still held cpuid = 12 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe124804f7a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850 vpanic() at vpanic+0x126/frame 0xfe124804f890 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920 vdropl() at vdropl+0x1c8/frame 0xfe124804f960 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 0xfe124804f9c0 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfe124804f9f0 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60 exit1() at exit1+0x541/frame 0xfe124804fad0 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 0xc45c, rbp = 0xc468 --- #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held)); (kgdb) print *lk $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 18446735288132049184, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96} Can you please grab people.freebsd.org/~jhb/gdb/* and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'? (kgdb) lockmgr_owner lk td: 0xf80272c61920 pid: 55040, p_comm: testprog Can you get a stack trace of that program? ('proc 55040', 'bt') -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On 3/3/2014 12:06 PM, John Baldwin wrote: On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote: On 2/28/2014 3:18 PM, John Baldwin wrote: On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote: While using poudriere: Unread portion of the kernel message buffer: panic: lockmgr still held cpuid = 12 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe124804f7a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850 vpanic() at vpanic+0x126/frame 0xfe124804f890 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920 vdropl() at vdropl+0x1c8/frame 0xfe124804f960 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 0xfe124804f9c0 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfe124804f9f0 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60 exit1() at exit1+0x541/frame 0xfe124804fad0 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 0xc45c, rbp = 0xc468 --- #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held)); (kgdb) print *lk $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 18446735288132049184, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96} Can you please grab people.freebsd.org/~jhb/gdb/* and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'? (kgdb) lockmgr_owner lk td: 0xf80272c61920 pid: 55040, p_comm: testprog Can you get a stack trace of that program? ('proc 55040', 'bt') It's the same as original: Loaded symbols for /boot/kernel/sem.ko.symbols #0 doadump (textdump=1) at pcpu.h:219 219 __asm(movq %%gs:%1,%0 : =r (td) (kgdb) bt #0 doadump (textdump=1) at pcpu.h:219 #1 0x808e33a7 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0x808e38b5 in vpanic (fmt=value optimized out, ap=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0x808e3749 in kassert_panic (fmt=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:647 #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 #5 0x80990d38 in vdropl (vp=0xf80a88a28588) at /usr/src/sys/kern/vfs_subr.c:2417 #6 0x80b73cbb in vm_object_deallocate (object=value optimized out) at /usr/src/sys/vm/vm_object.c:539 #7 0x80b6a3a9 in vm_map_process_deferred () at /usr/src/sys/vm/vm_map.c:2763 #8 0x80b6e9c8 in vm_map_remove (map=value optimized out, start=value optimized out, end=value optimized out) at /usr/src/sys/vm/vm_map.c:488 #9 0x80b6a179 in vmspace_exit (td=0xf80272c61920) at /usr/src/sys/vm/vm_map.c:327 #10 0x808a7411 in exit1 (td=0xf80272c61920, rv=value optimized out) at /usr/src/sys/kern/kern_exit.c:319 #11 0x808a6ece in sys_sys_exit (td=0x0, uap=value optimized out) at /usr/src/sys/kern/kern_exit.c:119 #12 0x80dd2720 in ia32_syscall (frame=0xfe124804fc00) at subr_syscall.c:133 #13 0x80cd6325 in Xint0x80_syscall () at ia32_exception.S:73 #14 0x281014df in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb) proc 55040 [Switching to thread 668 (Thread 100898)]#0 doadump (textdump=1) at pcpu.h:219 219 __asm(movq %%gs:%1,%0 : =r (td) (kgdb) bt #0 doadump (textdump=1) at pcpu.h:219 #1 0x808e33a7 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:452 #2 0x808e38b5 in vpanic (fmt=value optimized out, ap=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0x808e3749 in kassert_panic (fmt=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:647 #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 #5 0x80990d38 in vdropl (vp=0xf80a88a28588) at /usr/src/sys/kern/vfs_subr.c:2417 #6 0x80b73cbb in vm_object_deallocate (object=value optimized out) at /usr/src/sys/vm/vm_object.c:539 #7 0x80b6a3a9 in vm_map_process_deferred () at /usr/src/sys/vm/vm_map.c:2763 #8 0x80b6e9c8 in vm_map_remove (map=value optimized out, start=value optimized out, end=value optimized out) at /usr/src/sys/vm/vm_map.c:488 #9 0x80b6a179 in vmspace_exit (td=0xf80272c61920) at /usr/src/sys/vm/vm_map.c:327 #10 0x808a7411 in exit1
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On 2/28/2014 3:18 PM, John Baldwin wrote: On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote: While using poudriere: Unread portion of the kernel message buffer: panic: lockmgr still held cpuid = 12 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe124804f7a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850 vpanic() at vpanic+0x126/frame 0xfe124804f890 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920 vdropl() at vdropl+0x1c8/frame 0xfe124804f960 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 0xfe124804f9c0 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfe124804f9f0 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60 exit1() at exit1+0x541/frame 0xfe124804fad0 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 0xc45c, rbp = 0xc468 --- #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held)); (kgdb) print *lk $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 18446735288132049184, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96} Can you please grab people.freebsd.org/~jhb/gdb/* and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'? (kgdb) lockmgr_owner lk td: 0xf80272c61920 pid: 55040, p_comm: testprog -- Regards, Bryan Drewery signature.asc Description: OpenPGP digital signature
Re: panic: lockmgr still held [tmpfs] [vm_map_remove()-vdropl()] (r262186: Thu Feb 20)
On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote: While using poudriere: Unread portion of the kernel message buffer: panic: lockmgr still held cpuid = 12 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe124804f7a0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfe124804f850 vpanic() at vpanic+0x126/frame 0xfe124804f890 kassert_panic() at kassert_panic+0x139/frame 0xfe124804f900 lockdestroy() at lockdestroy+0x3b/frame 0xfe124804f920 vdropl() at vdropl+0x1c8/frame 0xfe124804f960 vm_object_deallocate() at vm_object_deallocate+0x10b/frame 0xfe124804f9c0 vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 0xfe124804f9f0 vm_map_remove() at vm_map_remove+0xc8/frame 0xfe124804fa20 vmspace_exit() at vmspace_exit+0xc9/frame 0xfe124804fa60 exit1() at exit1+0x541/frame 0xfe124804fad0 sys_sys_exit() at sys_sys_exit+0xe/frame 0xfe124804fae0 ia32_syscall() at ia32_syscall+0x270/frame 0xfe124804fbf0 Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfe124804fbf0 --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 0xc45c, rbp = 0xc468 --- #4 0x808c00db in lockdestroy (lk=0xf80a88a285f0) at /usr/src/sys/kern/kern_lock.c:440 440 KASSERT(lk-lk_lock == LK_UNLOCKED, (lockmgr still held)); (kgdb) print *lk $1 = {lock_object = {lo_name = 0x8201a1bd tmpfs, lo_flags = 116588552, lo_data = 0, lo_witness = 0xfe6fec00}, lk_lock = 18446735288132049184, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96} Can you please grab people.freebsd.org/~jhb/gdb/* and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 'lockmgr_owner lk'? -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org