Kahlil Oppenheimer created HBASE-17707:
------------------------------------------
Summary: New More Accurate TableSkew Balancer/Generator
Key: HBASE-17707
URL: https://issues.apache.org/jira/browse/HBASE-17707
Project: HBase
Issue Type: New Feature
Components: Balancer
Affects Versions: 1.2.0
Environment: CentOS Derivative with a derivative of the 3.18.43
kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches.
Reporter: Kahlil Oppenheimer
Priority: Minor
Attachments: HBASE-17707.patch
This patch includes new version of the TableSkewCostFunction and a new
TableSkewCandidateGenerator.
The new TableSkewCostFunction computes table skew by counting the minimal
number of region moves required for a given table to perfectly balance the
table across the cluster (i.e. as if the regions from that table had been
round-robin-ed across the cluster). This number of moves is computer for each
table, then normalized to a score between 0-1 by dividing by the number of
moves required in the absolute worst case (i.e. the entire table is stored on
one server), and stored in an array. The cost function then takes a weighted
average of the average and maximum value across all tables. The weights in this
average are configurable to allow for certain users to more strongly penalize
situations where one table is skewed versus where every table is a little bit
skewed. To better spread this value more evenly across the range 0-1, we take
the square root of the weighted average to get the final value.
The new TableSkewCandidateGenerator generates region moves/swaps to optimize
the above TableSkewCostFunction. It first simply tries to move regions until
each server has the right number of regions, then it swaps regions around such
that each region swap improves table skew across the cluster.
We tested the cost function and generator in our production clusters with 100s
of TBs of data and 100s of tables across dozens of servers and found both to be
very performant and accurate.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
