[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613401#comment-13613401
]
rajeshbabu commented on HBASE-8127:
-----------------------------------
bq. Will that work for you to consolidate the code into SSH?
I also got the same idea, but with this also what ever problem i told before
will happen.
Let us take an example. There are 250 regions in RIT need to process during
initialization and 250 regions of DISABLED table are opened before a server
went down. In that case SSH will take same time as master initialization
becuase in both the cases we need to read from zk.
IMHO, handling like is fine because we are sure that znode is present for a
disabling/disabled table region,transitioning on dead server.
{code}
if (actualDeadServers.contains(deadServer.getKey())) {
for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
+ HRegionInfo hri = deadRegion.getFirst();
+ // Delete znode of region in transition if table is disabled or
disabling. If a region
+ // server went down during master initialization then SSH cannot
handle the regions of
+ // partially disabled tables because in memory region state
information may not be
+ // available with master.
+ if
(zkTable.isDisablingOrDisabledTable(hri.getTableNameAsString())) {
+ try {
+ // If table is partially disabled then delete znode if exists
in any state.
+ ZKAssign.deleteNodeFailSilent(this.master.getZooKeeper(), hri);
+ } catch (KeeperException ke) {
+ this.master.abort("Unexpected ZK exception deleting unassigned
node " + hri, ke);
+ }
+ regionOffline(hri);
+ }
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
do you see any problems with this Jeffrey?
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.5
> Reporter: Jeffrey Zhong
> Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_feedback.patch,
> HBASE-8127.patch, hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
