[
https://issues.apache.org/jira/browse/HBASE-20634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16501218#comment-16501218
]
stack commented on HBASE-20634:
-------------------------------
bq. stack Have you seen my last comments on rb?
I did, just now. Thanks. I put a question up on rb but also opening a new issue
on what to do when src and target when moving a region are not online. Will
make subtask on this.
bq. I think the root cause is that there is no straight-forward rule for
fencing, and we add checks everywhere makes the code too complicated and hard
to understand...
Yeah.
Lock is 'fencing' at region level when only regions involved. We can't take a
table or server lock. The system would grind to a halt and not be 'live'. All
action is over RPC which can fail in an infinity of ways. SCP is 'repair' so
tries to override all procedures. It is arbiter over when regions can start
moving again (post-log-split). Procedures are written in different styles.
Would be fun to do a review, revamp. Any ideas?
What we have is better than what was there before (though still plenty
unknowns). At least you can reason about it this time around. Should move
faster too... which we need scaling.
> Reopen region while server crash can cause the procedure to be stuck
> --------------------------------------------------------------------
>
> Key: HBASE-20634
> URL: https://issues.apache.org/jira/browse/HBASE-20634
> Project: HBase
> Issue Type: Bug
> Reporter: Duo Zhang
> Assignee: stack
> Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20634-UT.patch, HBASE-20634.branch-2.0.001.patch,
> HBASE-20634.branch-2.0.002.patch, HBASE-20634.branch-2.0.003.patch,
> HBASE-20634.branch-2.0.004.patch, HBASE-20634.branch-2.0.005.patch,
> HBASE-20634.branch-2.0.006.patch, HBASE-20634.branch-2.0.006.patch,
> HBASE-20634.branch-2.0.007.patch, HBASE-20634.branch-2.0.008.patch,
> HBASE-20634.branch-2.0.009.patch
>
>
> Found this when implementing HBASE-20424, where we will transit the peer sync
> replication state while there is server crash.
> The problem is that, in ServerCrashAssign, we do not have the region lock, so
> it is possible that after we call handleRIT to clear the existing
> assign/unassign procedures related to this rs, and before we schedule the
> assign procedures, it is possible that that we schedule a unassign procedure
> for a region on the crashed rs. This procedure will not receive the
> ServerCrashException, instead, in addToRemoteDispatcher, it will find that it
> can not dispatch the remote call and then a FailedRemoteDispatchException
> will be raised. But we do not treat this exception the same with
> ServerCrashException, instead, we will try to expire the rs. Obviously the rs
> has already been marked as expired, so this is almost a no-op. Then the
> procedure will be stuck there for ever.
> A possible way to fix it is to treat FailedRemoteDispatchException the same
> with ServerCrashException, as it will be created in addToRemoteDispatcher
> only, and the only reason we can not dispatch a remote call is that the rs
> has already been dead. The nodeMap is a ConcurrentMap so I think we could use
> it as a guard.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
