[
https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116032#comment-13116032
]
Jonathan Gray commented on HBASE-4497:
--------------------------------------
I was just discussing this scenario with Dhruba a few days back. There's
definitely a race condition here and I don't see a trivial fix.
We use HLog IO-fencing to ensure that edits don't slip into an HLog after a
server is considered dead by the Master. But the Master has no way to prevent
this META update from slipping in.
We need to make some modification to how the master can safely timeout an
OPENING. One possibility is for the master to require either an acknowledgment
from the RS before moving the region elsewhere or for the RS to die. It seems
unlikely that we will actually see the RS to Master acknowledgment since
OPENING taking too long is usually a sign of brokenness or the RS being backed
up, I think. But in any case I'd imagine some kind of OPEN_CANCEL_REQUESTED
state that the Master transitions the node to and only when the RS transitions
to OPEN_CANCELED or OFFLINE or something, then it's safe to reassign elsewhere.
I think this design still has a hole in it though because there are scenarios
where the RS doesn't actually die but for some reason doesn't OPEN or ack the
cancel.
Another option would be to do the RS performed META edits using a CheckAndPut
rather than straight Put. Or we could move META editing back to the Master
where it's easy to do things atomically :)
The CheckAndPut idea is kind of neat but we'd probably have to send more data
on the OPEN_RPC. For example, the existing server start code or server name +
start code or something guaranteed unique (guaranteed that a conflicting RS
opening stuff wouldn't be able to use the same thing). Then the atomicity is
on the META region.
> If region opening fails after updating META HBCK reports it as inconsistent
> and scanning the region throws NSRE
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-4497
> URL: https://issues.apache.org/jira/browse/HBASE-4497
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Priority: Critical
>
> As per the discussion in the mail chain "HBCK reporting of possible mismatch
> in RS assignment" this JIRA is created.
> Consider two RS-> RS1 and RS2.
> A region tries to open in RS1. But it takes a while. The RS1 has still not
> updated meta and transitioned the node from OPENING to OPENED
> So timeout assigns the region to RS2. RS2 successfully updates the META and
> opens the region.
> Now RS1 tries to act on the region by first updating the META and then
> transiting the node to OPENING to OPENED.
> RS1 transiting the node to OPENING to OPENED will fail. But the META entry
> will have RS1 as the latest.
> Now HBCK reports this as an inconsistency and if we try to scan the Region we
> get NotServingRegionException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
