Yingchun Lai has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/18454 )
Change subject: [master] KUDU-3390 support auto rebalance tablet leaders across
TServers
......................................................................
[master] KUDU-3390 support auto rebalance tablet leaders across TServers
The number of leader replicas per tablet server can become imbalanced
over time, which lead to load skew on some nodes. This patch adds
auto leader rebalance task to avoid leader replicas skew.
Two reasons of load skew:
- The main reason. Scan Requests has two modes: LeaderOnly(default) and
CLOSEST_REPLICA.
For more accurate results, users will choose the LeaderOnly(default) mode.
Mostly, the scan load is positive correlation with leader numbers.
- The other reason. Write requests, leaders receive write requests and
followers receive
UpdateConsensus request(raft called AppendEntries), the flow of processing is a
little
different: the queue, pipeline, batch size, that may cause imbalanced load.
Leader
rebalance will make leader and followers balanced and make service more stable.
An experiment present leader rebalance's effect (more details at KUDU-3390):
|leader ratio| scan cost | cpu usage | io
before|40: 0: 0 | 811.586 s | 47%, 18%, 19% | 102MB/s ioutil:55%, 8KB/s
2%, 64KB/s 3%
after |13: 14: 13 | 611.012 s | 39%, 45%, 35% | 53MB/s ioutil:31%, 80MB/s
18%, 45MB/s 24%
Change-Id: Ibfb60d8759a93b6a19238637c27df4f6b1cac918
Reviewed-on: http://gerrit.cloudera.org:8080/18454
Reviewed-by: Yuqi Du <[email protected]>
Reviewed-by: Yingchun Lai <[email protected]>
Tested-by: Kudu Jenkins
---
M src/kudu/master/CMakeLists.txt
A src/kudu/master/auto_leader_rebalancer-test.cc
A src/kudu/master/auto_leader_rebalancer.cc
A src/kudu/master/auto_leader_rebalancer.h
M src/kudu/master/auto_rebalancer-test.cc
M src/kudu/master/auto_rebalancer.h
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
8 files changed, 836 insertions(+), 15 deletions(-)
Approvals:
Yuqi Du: Looks good to me, but someone else must approve
Yingchun Lai: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/18454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibfb60d8759a93b6a19238637c27df4f6b1cac918
Gerrit-Change-Number: 18454
Gerrit-PatchSet: 40
Gerrit-Owner: Yuqi Du <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Yifan Zhang <[email protected]>
Gerrit-Reviewer: Yingchun Lai <[email protected]>
Gerrit-Reviewer: Yuqi Du <[email protected]>
