On Mon, Nov 05, 2007 at 02:22:12PM +0100, Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm observing the following deadlock.
>
> One thread holds zfsvfs->z_hold_mtx[i] lock and waits for I/O:
[...]
> Another thread tries to finish I/O, but can't, because it is trying to
> acquire zfsvfs->z_hold_mtx[i] lock:
[...]
> As you can see, the SPA ZIO INTR thread wants to finish I/O requests, calls
> arc_write_done(). All in all we call vn_rele(), which may end up in calling
> zfs_zinactive() if this was the last vnode reference. zfs_zinactive() tries
> to acquire zfsvfs->z_hold_mtx[i] lock, but can't, because it is already held
> by the thread waiting for I/O. Neither I/O can be finished, nor lock can be
> unlocked.
>
> I started to see this deadlock after last integration, so it was introduced
> in more or less during last month. Any ideas how to solve it?
I was able to trigger this deadlock with older code as well.
It seems I fixed it be changing the zfs_get_done() function from:
void
zfs_get_done(dmu_buf_t *db, void *vzgd)
{
zgd_t *zgd = (zgd_t *)vzgd;
rl_t *rl = zgd->zgd_rl;
vnode_t *vp = ZTOV(rl->r_zp);
dmu_buf_rele(db, vzgd);
zfs_range_unlock(rl);
VN_RELE(vp);
zil_add_vdev(zgd->zgd_zilog, DVA_GET_VDEV(BP_IDENTITY(zgd->zgd_bp)));
kmem_free(zgd, sizeof (zgd_t));
}
To something like this:
void
zfs_get_done(dmu_buf_t *db, void *vzgd)
{
zgd_t *zgd = (zgd_t *)vzgd;
rl_t *rl = zgd->zgd_rl;
vnode_t *vp = ZTOV(rl->r_zp);
dmu_buf_rele(db, vzgd);
zfs_range_unlock(rl);
zgd->zgd_vnode = vp;
zil_add_vdev(zgd->zgd_zilog, DVA_GET_VDEV(BP_IDENTITY(zgd->zgd_bp)));
/*
* Put the vnode onto the list. Call VN_RELE() on this vnode from
* within another kernel thread later (also free zgd there).
*/
mtx_lock(&zgd_done_mtx);
TAILQ_INSERT_TAIL(&zgd_done, zgd, zgd_next);
mtx_unlock(&zgd_done_mtx);
wakeup(&zgd_done);
}
Which bascially means, that VN_RELE() will be called from another thread
later. This seems to work, but I changed the order of VN_RELE() and
zil_add_vdev() calls. Is that safe? Doing zil_add_vdev() also from this
additional thread didn't worked for me.
--
Pawel Jakub Dawidek http://www.wheel.pl
pjd at FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-code/attachments/20071107/e27a4697/attachment.bin>
