[
https://issues.apache.org/jira/browse/HBASE-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149042#comment-15149042
]
Clara Xiong commented on HBASE-15251:
-------------------------------------
[~syuanjiang] This patch is what we are running in production. Let me answer
your questions:
1. I agree dead server with 0 region assigned could happen in failover. So
after this check set failover to false, my code continues on to all the
remaining checks: region in transition, unprocessed wal files and etc. If and
only if all the other checks also set failover to false, we continue to the
clean restart route. This is consistent with existing logic that if everything
is in clean state, we can treat the situation as a clean restart. The
additional check on dead server I added was to avoid declare failover
unnecessarily.
2. The improvement we have seen was from the difference of clean restart route
and failover route, not from fewer CPU cycles doing fewer checks. Clean restart
doesn't need to reassign the regions. We have > 1,000 nodes and > 50,000
regions, the performance gain is very significant.
> During a cluster restart, Hmaster thinks it is a failover by mistake
> --------------------------------------------------------------------
>
> Key: HBASE-15251
> URL: https://issues.apache.org/jira/browse/HBASE-15251
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.0.0, 0.98.15
> Reporter: Clara Xiong
> Assignee: Clara Xiong
> Attachments: HBASE-15251-master.patch
>
>
> We often need to do cluster restart as part of release for a cluster of >
> 1000 nodes. We have tried our best to get clean shutdown but 50% of the time,
> hmaster still thinks it is a failover. This increases the restart time from 5
> min to 30 min and decreases locality from 99% to 5% since we didn't use a
> locality-aware balancer. We had a bug HBASE-14129 but the fix didn't work.
> After adding more logging and inspecting the logs, we identified two things
> that trigger the failover handling:
> 1. When Hmaster.AssignmentManager detects any dead servers on service
> manager during joinCluster(), it determines this is a failover without
> further check. I added a check whether there is even any region assigned to
> these servers. During a clean restart, the regions are not even assigned.
> 2. When there are some leftover empty folders for log and split directories
> or empty wal files, it is also treated as a failover. I added a check for
> that. Although this can be resolved by manual cleanup, it is still too
> tedious for restarting a large cluster.
> Patch will follow shortly. The fix is tested and used in production now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
