Alexey Serbin has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/18294 )
Change subject: [tools] range rebalancing for 'kudu cluster rebalance'
......................................................................
[tools] range rebalancing for 'kudu cluster rebalance'
This patch adds range rebalancing functionality into the
'kudu cluster rebalance' CLI tool. The implementation is rather an
MVP: the range rebalancing can now be performed only for a single
table per run. As far as I can see, there is a room for improvement
since it's possible to perform range-aware replica movements even
during standard whole cluster rebalancing.
Below is two snapshots of distribution of the range-specific tablet
replicas in a cluster. Those are produced by running the tool with
extra --report_only --output_replica_distribution_details flags
before and after range rebalancing for a single table:
kudu cluster rebalance \
--enable_range_rebalancing \
--tables=default.loadgen_auto_6800f4ec4e164b2b8e42db7b5044df09 \
127.0.0.1:8765
before:
========================================================================
Table: abb2bbf8b4ff4bc0989bc82c78d4ae2b
Number of tablet replicas at servers for each range
Max Skew | Total Count | Range Start Key
----------+-------------+----------------------
8 | 8 |
8 | 8 | ff80000000000001fff4
8 | 8 | ff80000000000003ffe8
8 | 8 | ff80000000000005ffdc
Range start key: ''
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 8
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000001fff4'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 8
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000003ffe8'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 8
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000005ffdc'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 8
after:
========================================================================
Table: abb2bbf8b4ff4bc0989bc82c78d4ae2b
Number of tablet replicas at servers for each range
Max Skew | Total Count | Range Start Key
----------+-------------+----------------------
0 | 8 |
0 | 8 | ff80000000000001fff4
0 | 8 | ff80000000000003ffe8
0 | 8 | ff80000000000005ffdc
Range start key: ''
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000001fff4'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000003ffe8'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000005ffdc'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Change-Id: I7d2e19266e993f5e2ae13ba18d323c83db30eac1
Reviewed-on: http://gerrit.cloudera.org:8080/18294
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Andrew Wong <[email protected]>
---
M src/kudu/master/auto_rebalancer.cc
M src/kudu/rebalance/rebalance-test.cc
M src/kudu/rebalance/rebalance_algo-test.cc
M src/kudu/rebalance/rebalance_algo.cc
M src/kudu/rebalance/rebalance_algo.h
M src/kudu/rebalance/rebalancer.cc
M src/kudu/rebalance/rebalancer.h
M src/kudu/tools/rebalancer_tool-test.cc
M src/kudu/tools/rebalancer_tool.cc
M src/kudu/tools/tool_action_cluster.cc
10 files changed, 1,233 insertions(+), 329 deletions(-)
Approvals:
Alexey Serbin: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/18294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7d2e19266e993f5e2ae13ba18d323c83db30eac1
Gerrit-Change-Number: 18294
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
