[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612397#comment-13612397
]
Jeffrey Zhong commented on HBASE-8127:
--------------------------------------
[~rajesh23] I reviewed your latest path with one question:
{code}
+ HRegionInfo hri = deadRegion.getFirst();
+ // Delete znode of region in transition if table is disabled or
disabling. If a region
+ // server went down during master initialization then SSH cannot
handle the regions of
+ // partially disabled tables because in memory region state
information may not be
+ // available with master.
+ if
(zkTable.isDisablingOrDisabledTable(hri.getTableNameAsString())) {
+ try {
+ // If table is partially disabled then delete znode if exists
in any state.
+ ZKAssign.deleteNodeFailSilent(this.master.getZooKeeper(), hri);
+ } catch (KeeperException ke) {
+ this.master.abort("Unexpected ZK exception deleting unassigned
node " + hri, ke);
+ }
+ regionOffline(hri);
+ }
{code}
Why don't you handle this situation in SSH as well by removing RIT null check?
There are two following reasons we should put in SSH:
1) another RS with similar states could fail right after this check(RITs may
still not be fully initialized)
2) In SSH, you only check RIT nullness not values, therefore, it's safe to
remove the check because the null states should only happen in master situation
phase. Any issue do you see when remove the RIT null check?
3) In SSH, we remove a region from assignment while may still leave RIT in ZK
due to the null check. So it's possible some RIT could stuck there forever.
The rest path looks good to me.
Thanks,
-Jeffrey
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.5
> Reporter: Jeffrey Zhong
> Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_feedback.patch,
> HBASE-8127.patch, hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
