[
https://issues.apache.org/jira/browse/HBASE-22349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-22349:
----------------------------------
Description:
HBASE-24139 allows the load balancer to run when one server has 0 regions and
another server has more than 1 region. This is a special case of a more generic
problem, where one server has far too few or far too many regions. The
StochasticLoadBalancer defaults may decide the cluster is "balanced enough"
according to {{hbase.master.balancer.stochastic.minCostNeedBalance}}, even
though one server may have a far higher or lower number of regions compared to
the rest of the cluster.
One specific example of this we have seen is when we use {{RegionMover}} to
move regions back to a restarted RegionServer, if the
{{StochasticLoadBalancer}} happens to be running. The load balancer sees a
newly restarted RegionServer with 0 regions, and after HBASE-24139, it will
balance regions to this server. Simultaneously, {{RegionMover}} moves back
regions. The end result is that the newly restarted RegionServer has twice the
load of any other server in the cluster. Future iterations of the load balancer
do nothing, as the cluster cost does not exceed {{minCostNeedBalance}}.
Another example is if the load balancer makes very slow progress on a cluster,
it may not move the average cluster load to a newly restarted regionserver in
one iteration. But after the first iteration, the balancer may again not run
due to cluster cost not exceeding {{minCostNeedBalance}}.
We can propose a solution where we reuse the {{slop}} concept in
{{SimpleLoadBalancer}} and use this to extend the HBASE-24139 logic for
deciding to run the balancer as long as there is a "sloppy" server in the
cluster.
+*Previous Description Notes Below, which are relevant, but as stated, were
already fixed by HBASE-24139*+
In EMR cluster, whenever I replace one of the nodes, the regions never get
rebalanced.
The default minCostNeedBalance set to 0.05 is too high.
The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20 =
203
Once a node(region server) got replaced with a new node (terminated and EMR
recreated a node), the region count on the servers became: 23, 0, 23, 22, 22,
22, 22, 23, 23, 23 = 203
>From hbase-master-logs, I can see the below WARN which indicates that the
>default minCostNeedBalance does not hold good for these scenarios.
##
2019-04-29 09:31:37,027 WARN
[ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
cleaner.CleanerChore: WALs outstanding under
hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29
09:31:42,920 INFO
[ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
balancer.StochasticLoadBalancer: Skipping load balancing because balanced
cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost
which need balance is 0.05
##
To mitigate this, I had to modify the default minCostNeedBalance to lower value
like 0.01f and restart Region Servers and Hbase Master. After modifying this
value to 0.01f I could see the regions getting re-balanced.
This has led me to the following questions which I would like to get it
answered from the HBase experts.
1)What are the factors that affect the value of total cost and sum multiplier?
How could we determine the right minCostNeedBalance value for any cluster?
2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal
value? If yes, then what is the recommended way to mitigate this scenario?
Attached: Steps to reproduce
Note: HBase-17565 patch is already applied.
was:
In EMR cluster, whenever I replace one of the nodes, the regions never get
rebalanced.
The default minCostNeedBalance set to 0.05 is too high.
The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20 =
203
Once a node(region server) got replaced with a new node (terminated and EMR
recreated a node), the region count on the servers became: 23, 0, 23, 22, 22,
22, 22, 23, 23, 23 = 203
>From hbase-master-logs, I can see the below WARN which indicates that the
>default minCostNeedBalance does not hold good for these scenarios.
##
2019-04-29 09:31:37,027 WARN
[ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
cleaner.CleanerChore: WALs outstanding under
hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29
09:31:42,920 INFO
[ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
balancer.StochasticLoadBalancer: Skipping load balancing because balanced
cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost
which need balance is 0.05
##
To mitigate this, I had to modify the default minCostNeedBalance to lower value
like 0.01f and restart Region Servers and Hbase Master. After modifying this
value to 0.01f I could see the regions getting re-balanced.
This has led me to the following questions which I would like to get it
answered from the HBase experts.
1)What are the factors that affect the value of total cost and sum multiplier?
How could we determine the right minCostNeedBalance value for any cluster?
2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal
value? If yes, then what is the recommended way to mitigate this scenario?
Attached: Steps to reproduce
Note: HBase-17565 patch is already applied.
> Stochastic Load Balancer skips balancing when node is replaced in cluster
> -------------------------------------------------------------------------
>
> Key: HBASE-22349
> URL: https://issues.apache.org/jira/browse/HBASE-22349
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha-1, 1.3.0, 1.4.4, 2.0.0
> Reporter: Suthan Phillips
> Assignee: David Manning
> Priority: Major
> Attachments: Hbase-22349.pdf
>
>
> HBASE-24139 allows the load balancer to run when one server has 0 regions and
> another server has more than 1 region. This is a special case of a more
> generic problem, where one server has far too few or far too many regions.
> The StochasticLoadBalancer defaults may decide the cluster is "balanced
> enough" according to {{hbase.master.balancer.stochastic.minCostNeedBalance}},
> even though one server may have a far higher or lower number of regions
> compared to the rest of the cluster.
> One specific example of this we have seen is when we use {{RegionMover}} to
> move regions back to a restarted RegionServer, if the
> {{StochasticLoadBalancer}} happens to be running. The load balancer sees a
> newly restarted RegionServer with 0 regions, and after HBASE-24139, it will
> balance regions to this server. Simultaneously, {{RegionMover}} moves back
> regions. The end result is that the newly restarted RegionServer has twice
> the load of any other server in the cluster. Future iterations of the load
> balancer do nothing, as the cluster cost does not exceed
> {{minCostNeedBalance}}.
> Another example is if the load balancer makes very slow progress on a
> cluster, it may not move the average cluster load to a newly restarted
> regionserver in one iteration. But after the first iteration, the balancer
> may again not run due to cluster cost not exceeding {{minCostNeedBalance}}.
> We can propose a solution where we reuse the {{slop}} concept in
> {{SimpleLoadBalancer}} and use this to extend the HBASE-24139 logic for
> deciding to run the balancer as long as there is a "sloppy" server in the
> cluster.
> +*Previous Description Notes Below, which are relevant, but as stated, were
> already fixed by HBASE-24139*+
> In EMR cluster, whenever I replace one of the nodes, the regions never get
> rebalanced.
> The default minCostNeedBalance set to 0.05 is too high.
> The region count on the servers were: 21, 21, 20, 20, 20, 20, 21, 20, 20, 20
> = 203
> Once a node(region server) got replaced with a new node (terminated and EMR
> recreated a node), the region count on the servers became: 23, 0, 23, 22, 22,
> 22, 22, 23, 23, 23 = 203
> From hbase-master-logs, I can see the below WARN which indicates that the
> default minCostNeedBalance does not hold good for these scenarios.
> ##
> 2019-04-29 09:31:37,027 WARN
> [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
> cleaner.CleanerChore: WALs outstanding under
> hdfs://ip-172-31-35-122.ec2.internal:8020/user/hbase/oldWALs2019-04-29
> 09:31:42,920 INFO
> [ip-172-31-35-122.ec2.internal,16000,1556524892897_ChoreService_1]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 52.041826194833405, sum multiplier is 1102.0 min cost
> which need balance is 0.05
> ##
> To mitigate this, I had to modify the default minCostNeedBalance to lower
> value like 0.01f and restart Region Servers and Hbase Master. After modifying
> this value to 0.01f I could see the regions getting re-balanced.
> This has led me to the following questions which I would like to get it
> answered from the HBase experts.
> 1)What are the factors that affect the value of total cost and sum
> multiplier? How could we determine the right minCostNeedBalance value for any
> cluster?
> 2)How did Hbase arrive at setting the default value to 0.05f? Is it optimal
> value? If yes, then what is the recommended way to mitigate this scenario?
> Attached: Steps to reproduce
>
> Note: HBase-17565 patch is already applied.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
