On Mon, Jun 09, 2014 at 01:04:00PM -0700, Andrew Morton wrote:
> From: Tariq Saeed <tariq.x.sa...@oracle.com>
> Subject: ocfs2: fix deadlock when two nodes are converting same lock from PR
> to EX and idletimeout closes conn
>
> Orabug: 18639535
>
> Two node cluster and both nodes hold a lock at PR level and both want to
> convert to EX at the same time. Master node 1 has sent BAST and then
> closes the connection due to idletime out. Node 0 receives BAST, sends
> unlock req with cancel flag but gets error -ENOTCONN. The problem is this
> error is ignored in dlm_send_remote_unlock_request() on the **incorrect**
> assumption that the master is dead. See NOTE in comment why it returns
> DLM_NORMAL. Upon getting DLM_NORMAL, node 0 proceeds to sends convert
> (without cancel flg) which fails with -ENOTCONN. waits 5 sec and resends.
> This time gets DLM_IVLOCKID from the master since lock not found in grant
> , it had been moved to converting queue in response to conv PR->EX req.
> No way out.
>
> Node 1 (master) Node 0
> ============== ======
>
> lock mode PR PR
>
> convert PR -> EX
> mv grant -> convert and que BAST
> ...
> <-------- convert PR -> EX
> convert que looks like this: ((node 1, PR -> EX) (node 0, PR -> EX))
> ...
> BAST (want PR -> NL)
> ------------------>
> ...
> idle timout, conn closed
> ...
> In response to BAST,
> sends unlock with cancel convert flag
> gets -ENOTCONN. Ignores and
> sends remote convert request
> gets -ENOTCONN, waits 5 Sec, retries
> ...
> reconnects
> <----------------- convert req goes through on next try
> does not find lock on grant que
> status DLM_IVLOCKID
> ------------------>
> ...
>
> No way out. Fix is to keep retrying unlock with cancel flag until it
> succeeds or the master dies.
>
> Signed-off-by: Tariq Saeed <tariq.x.sa...@oracle.com>
> Cc: Mark Fasheh <mfas...@suse.com>
> Cc: Joel Becker <jl...@evilplan.org>
> Signed-off-by: Andrew Morton <a...@linux-foundation.org>
That looks good, thanks Tariq, in particular for the detailed explanation.
Reviewed-by: Mark Fasheh <mfas...@suse.de>
--Mark
--
Mark Fasheh
_______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel@oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel
