[
https://issues.apache.org/jira/browse/HBASE-30445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-30445:
----------------------------------
Status: Patch Available (was: In Progress)
> (branch-2) RegionStates may has some expired serverinfo and make regions do
> not balance.
> ----------------------------------------------------------------------------------------
>
> Key: HBASE-30445
> URL: https://issues.apache.org/jira/browse/HBASE-30445
> Project: HBase
> Issue Type: Bug
> Components: Balancer
> Affects Versions: 2.5.4, 2.6.0, 2.7.0
> Reporter: David Manning
> Assignee: David Manning
> Priority: Minor
> Labels: pull-request-available
>
> It is possible for the balancer to get into a state where it indefinitely
> seeks balance but cannot obtain it. LoadBalancer logs will show
> {{balancer.StochasticLoadBalancer - Running balancer because cluster has
> sloppy server(s).}}
> This happens indefinitely because the {{BalancerClusterState}} construction
> adds a key twice for the same server. From here, the balancer thinks there is
> always a server that has 0 regions, which is "sloppy", and forces a balancer
> run even if things are otherwise balanced. The logs may look like below
> (notice server 23 is listed twice):
> {{balancer.BalancerClusterState - server 24 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 23 is on rack 0}}
> {{balancer.BalancerClusterState - server 22 is on rack 0}}
> {{balancer.BalancerClusterState - server 21 is on rack 0}}
> The workaround is a restart of hmaster to clear this buggy state.
> The likely cause in this deployment, though not verified, was missing
> HBASE-29323. As a result, ReportRegionStateTransition RPCs could have been
> significantly delayed/deadlocked in the queue behind Move RPCs from the
> RegionMover, causing stale servers to reappear in the RegionStates.serverMap
> even (long) after their ServerCrashProcedure was complete. This is because
> {{{}AssignmentManager.reportRegionStateTransition }}has an unconditional call
> to \{{RegionStates.getOrCreateServer() }}which could execute after\{{
> ServerCrashProcedure }}calls\{{{} removeServer{}}}. Ultimately, this is the
> same type of cause described by HBASE-23564.
> However, after investigating the issue, 3.0 branch has deviated from 2.x
> branch in a way that appears unintentional. Because HBASE-23102 was
> backported later, it touched the line in an unexpected way, reverting back to
> the original code. The balancer should provide its baseline of available
> servers from {{onlineServers}} instead of {{{}serverMap.keySet(){}}}.
> This issue intends to bring 2.x branches back in line with 3.x branches.
> 3.0 branch history:
> # HBASE-22523 adds as {{{}serverMap.keySet(){}}}:
> [https://github.com/apache/hbase/commit/04e5bf96d84ff2f111b1e879c8daa5360b621922#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R552-R556]
>
> # HBASE-23102 does not touch the line :
> [https://github.com/apache/hbase/commit/a3efa5911d071a6848459b37a220d36177aec490#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R557]
>
> # HBASE-23564 changes to {{{}onlineServers{}}}:
> [https://github.com/apache/hbase/commit/ab40b9648b20cef6cb041789e4fbb11e4d36e3c2#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R562]
>
> 2.x branch history:
> # HBASE-22523 adds as {{{}serverMap.keySet(){}}}:
> [https://github.com/apache/hbase/commit/14041b39f2da17de7a654a1733f3fa136fe13f63#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R552-R556]
> # HBASE-23564 changes to {{{}onlineServers{}}}:
> [https://github.com/apache/hbase/commit/7a0e4d814059e6a26abc834770e5f66231e0400e#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R563]
>
> # HBASE-23102 changes back to {{{}serverMap.keySet(){}}}:
> [https://github.com/apache/hbase/commit/3b6d8c3394a22c71b9836c4000dbd582af248aa3#diff-96d9ec3583f3683e032c43a8f2561e06888bf1fe5eb4e818b3fae47b379ef196R585]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
