Re: Freebsd 11.0 RELEASE - ZFS deadlock
On 11/11/2016 12:24, Andriy Gapon wrote: At this stage I would try to get a system crash dump for post-mortem analysis. There are a few way to do that. You can enter ddb and then run 'dump' and 'reset' commands. Or you can just do `sysctl debug.kdb.panic=1`. In either case, please double-check that your system has a dump device configured. It take some time to upload the dump... You can find it at http://tignes.restart.be/Xfer/ Henri ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Freebsd 11.0 RELEASE - ZFS deadlock
On 10/11/2016 21:41, Henri Hennebert wrote:
> On 11/10/2016 19:40, Andriy Gapon wrote:
>> On 10/11/2016 19:55, Henri Hennebert wrote:
>>>
>>>
>>> On 11/10/2016 18:33, Andriy Gapon wrote:
On 10/11/2016 18:12, Henri Hennebert wrote:
> On 11/10/2016 16:54, Andriy Gapon wrote:
>> On 10/11/2016 17:20, Henri Hennebert wrote:
>>> On 11/10/2016 15:00, Andriy Gapon wrote:
Interesting. I can not spot any suspicious thread that would hold the
vnode
lock. Could you please run kgdb (just like that, no arguments), then
execute
'bt' command and then select a frame when _vn_lock is called with 'fr
N'
command. Then please 'print *vp' and share the result.
>>> I Think I miss something in your request:
>>
>> Oh, sorry! The very first step should be 'tid 101112' to switch to the
>> correct
>> context.
>>
>
> (kgdb) fr 7
> #7 0x8063c5b3 in _vn_lock (vp=,
> flags=2121728,
"value optimized out" - not good
> file=,
> line=) at vnode_if.h:859
> 859vnode_if.h: No such file or directory.
> in vnode_if.h
> (kgdb) print *vp
I am not sure if this output is valid, because of the message above.
Could you please try to navigate to nearby frames and see if vp itself has
a
valid value there. If you can find such a frame please do *vp there.
>>>
>>> Does this seems better?
>>
>> Yes!
>>
>>> (kgdb) fr 8
>>> #8 0x8062a5f7 in vget (vp=0xf80049c2c000, flags=2121728,
>>> td=0xf80009ba0500) at /usr/src/sys/kern/vfs_subr.c:2523
>>> 2523if ((error = vn_lock(vp, flags)) != 0) {
>>> (kgdb) print *vp
>>> $1 = {v_tag = 0x813be535 "zfs", v_op = 0x813d0f70, v_data =
>>> 0xf80049c1f420, v_mount = 0xf800093aa660,
>>> v_nmntvnodes = {tqe_next = 0xf80049c2c938, tqe_prev =
>>> 0xf80049c2bb30},
>>> v_un = {vu_mount = 0x0, vu_socket = 0x0,
>>> vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev
>>> =
>>> 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {
>>> tqh_first = 0xf800bfc8e3f0, tqh_last = 0xf800bfc8e410},
>>> v_cache_dd =
>>> 0x0, v_lock = {lock_object = {
>>> lo_name = 0x813be535 "zfs", lo_flags = 117112832, lo_data = 0,
>>> lo_witness = 0x0}, lk_lock = 23, lk_exslpfail = 0,
>>> lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name =
>>> 0x8099e9e0 "vnode interlock", lo_flags = 16973824,
>>> lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, v_vnlock =
>>> 0xf80049c2c068, v_actfreelist = {
>>> tqe_next = 0xf80049c2c938, tqe_prev = 0xf80049ae9bd0}, v_bufobj
>>> =
>>> {bo_lock = {lock_object = {
>>> lo_name = 0x8099e9f0 "bufobj interlock", lo_flags =
>>> 86179840,
>>> lo_data = 0, lo_witness = 0x0}, rw_lock = 1},
>>> bo_ops = 0x80c4bf70, bo_object = 0xf800b62e9c60,
>>> bo_synclist =
>>> {le_next = 0x0, le_prev = 0x0},
>>> bo_private = 0xf80049c2c000, __bo_vnode = 0xf80049c2c000,
>>> bo_clean =
>>> {bv_hd = {tqh_first = 0x0,
>>> tqh_last = 0xf80049c2c120}, bv_root = {pt_root = 0}, bv_cnt =
>>> 0},
>>> bo_dirty = {bv_hd = {tqh_first = 0x0,
>>> tqh_last = 0xf80049c2c140}, bv_root = {pt_root = 0}, bv_cnt =
>>> 0},
>>> bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072},
>>> v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters =
>>> {tqh_first = 0x0, tqh_last = 0xf80049c2c188},
>>> rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0,
>>> v_holdcnt = 9, v_usecount = 6, v_iflag = 512,
>>> v_vflag = 32, v_writecount = 0, v_hash = 4833984, v_type = VREG}
>>> (kgdb)
>>
>> flags=2121728 = 0x206000 = LK_SHARED | LK_VNHELD | LK_NODDLKTREAT
>> lk_lock = 23 = 0x17 = LK_ONE_SHARER | LK_EXCLUSIVE_WAITERS |
>> LK_SHARED_WAITERS |
>> LK_SHARE
>>
>> So, here's what we have here: this thread tries to get a shared lock on the
>> vnode, the vnode is already locked in shared mode, but there is an exclusive
>> waiter (or, perhaps, multiple waiters). So, this thread can not get the lock
>> because of the exclusive waiter. And I do not see an easy way to identify
>> that
>> waiter.
>>
>> In the procstat output that you provided earlier there was no other thread in
>> vn_lock. Hmm, I see this:
>> procstat: sysctl: kern.proc.kstack: 14789: Device busy
>> procstat: sysctl: kern.proc.kstack: 82034: Device busy
>>
>> Could you please check what those two processes are (if they are still
>> running)?
>> Perhaps try procstat for each of the pids several times.
>>
At this stage I would try to get a system crash dump for post-mortem analysis.
There are a few way to do that. You can enter ddb and then run 'dump' and
'reset' commands. Or you can just do `sysctl debug.kdb.panic=1`.
In either case, please double-check that your system has a dump device
configured.
> This 2 processes are the 2
FreeBSD_STABLE_10-i386 - Build #1505 - Fixed
FreeBSD_STABLE_10-i386 - Build #1505 - Fixed: Build information: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1505/ Full change log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1505/changes Full build log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/1505/console Change summaries: 308512 by sephe: MFC 308164 hyperv/hn: Regroup if_start related functions. And put them under HN_IFSTART_SUPPORT, which is by default on until we whack the if_start related bits from base system. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8392 308511 by sephe: MFC 308163 hyperv/hn: Rename cleaned up file. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8390 308510 by sephe: MFC 308162 hyperv/hn: Cosmetic cleanup; no functional changes. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8389 308509 by sephe: MFC 308117-308120 308117 hyperv/hn: Rework temporary channel packet buffer expanding. And use large default temporary channel packer buffer; we really don't want it to be expanded at run time. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8367 308118 hyperv/hn: Cleanup RXBUF ack processing. - Increase the # of retries. - Add comment. - Log error, if RXBUF ack fails. - Add stat for RXBUF ack failures. RXBUF ack really should _not_ fail... Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8368 308119 hyperv/hn: Reset do_lro, if the hash types are not TCP related. Mainly because the host side only set TCPCS and IPCS even for UDP datagrams. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8369 308120 hyperv/hn: Don't start shared TX taskq, if the hypervisor is not Hyper-V. - Move the SYSINIT to DRIVER/SECOND, i.e. after the vm_guest becomes determistic. - Minor style changes. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8370 308508 by sephe: MFC 308018,308116 308018 hyeprv/hn: Rename cleaned up RNDIS header file. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8360 308116 hyperv/hn: Rename cleaned up RNDIS source file. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8361 308507 by sephe: hyperv/hn: Fix i386 build; if_baudrate is 32bits on i386 on stable/10 This is a direct commit. Sponsored by: Microsoft 308506 by sephe: MFC 308013-308017 308013 hyperv/hn: Nuke unnecessary indirection. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8355 308014 hyperv/hn: Reorganize RX path; mainly pull non-control code path up Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8356 308015 hyperv/hn: Pull data path code up. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8357 308016 hyperv/hn: Cleanup RNDIS related files. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8358 308017 hyperv/hn: Change header guardian; in preparation for the upcoming rename. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8359 308505 by sephe: MFC 308011,308012 308011 hyperv/hn: Rename cleaned up NVS header file. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8353 308012 hyperv/hn: Rename cleaned up NVS source file. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8354 308504 by sephe: MFC 307989-307991,308010 307989 hyperv/hn: Move hn_softc to if_hnvar.h While I'm here, use consistent macro names. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8345 307990 hyperv/hn: Move send context to NVS domain. Since all sends are encapsulated in NVS messages. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8346 307991 hyperv/hn: NVS inclusion cleanup and forward declare functions. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8347 308010 hyperv/hn: Change header guardian; in preparation for the upcoming rename. Sponsored by: Microsoft Differential Revision: https://reviews.freebsd.org/D8352 ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
