[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613530#comment-13613530
]
Jeffrey Zhong commented on HBASE-8127:
--------------------------------------
The reason for me to raise the question is that we have very similar code in
two places. This will likely cause issue later if we need to change the logic
in the snippet of code because it's quite possible to forget to update both
places.
{quote}
Let us take an example. There are 250 regions in RIT need to process during
initialization and 250 regions of DISABLED table are opened before a server
went down. In that case SSH will take same time as master initialization
becuase in both the cases we need to read from zk.
{quote}
If it's a dead server, we will skip those regions by
{code}nodes.remove(deadRegion.getFirst().getEncodedName());{code} during
initialization and SSH is handed in a separate thread of a thread pool. So just
the thread which handles the dead server SSH will slow down not others. No
matter what the deletions have to happen and it's preferable not in master
initialization path. If we block master node by this, it has more impacts
because mater node is kind of single failure point and we need it start as fast
as possible. If there is no correctness issue, it's better to consolidate the
code in SSH. The concern you have can happen in normal cases more frequently
such as after the master starts up and a server dies with lots of RIT regions
of disabling tables. If the ZK operations still concern you, you can handle
them asynchronously while I think it's a little bit too much for this.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.5
> Reporter: Jeffrey Zhong
> Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_feedback.patch,
> HBASE-8127.patch, hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair<HRegionInfo, Result> deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
