Alexey Serbin has uploaded this change for review. (
http://gerrit.cloudera.org:8080/18294
Change subject: WIP: [tools] range rebalancing for 'kudu cluster rebalance'
......................................................................
WIP: [tools] range rebalancing for 'kudu cluster rebalance'
WIP:
* split into logically independent patches
* proper commit description
* clean-up TODO(s)
* add more docs
* add more tests
This patch adds range rebalancing functionality into the
'kudu cluster rebalance' CLI tool. The implementation is rather an
MVP: the range rebalancing can now be performed only for a single
table per run. As far as I can see, there is a room for improvement
since it's possible to perform range-aware replica movements even
during standard whole cluster rebalancing.
Below is two snapshots of distribution of the range-specific tablet
replicas in a cluster. Those are produced by running the tool with
extra --report_only --output_replica_distribution_details flags
before and after range rebalancing for a single table:
kudu cluster rebalance \
--enable_range_rebalancing \
--tables=default.loadgen_auto_6800f4ec4e164b2b8e42db7b5044df09 \
127.0.0.1:8765
before:
========================================================================
Table: abb2bbf8b4ff4bc0989bc82c78d4ae2b
Range start key: ''
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 8
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000001fff4'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 8
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000003ffe8'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 8
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 0
Range start key: 'ff80000000000005ffdc'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 0
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 0
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 0
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 8
after:
========================================================================
Table: abb2bbf8b4ff4bc0989bc82c78d4ae2b
Range start key: ''
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000001fff4'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000003ffe8'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Range start key: 'ff80000000000005ffdc'
UUID | Server address | Replica Count
----------------------------------+----------------+---------------
15a8d0fef42c4da2bd5d9e1c5a2de301 | 127.0.0.1:9870 | 2
3243029e0db04680a2653c6acc048813 | 127.0.0.1:9876 | 2
324ef10666b14ab9bb61e775fa351ad6 | 127.0.0.1:9872 | 2
a5c6f822f5cc4645bbb4a14874e311d4 | 127.0.0.1:9874 | 2
Change-Id: I7d2e19266e993f5e2ae13ba18d323c83db30eac1
---
M src/kudu/client/CMakeLists.txt
M src/kudu/client/client.h
A src/kudu/client/tablet_info_provider-internal.cc
A src/kudu/client/tablet_info_provider-internal.h
M src/kudu/rebalance/cluster_status.h
M src/kudu/rebalance/rebalance-test.cc
M src/kudu/rebalance/rebalance_algo-test.cc
M src/kudu/rebalance/rebalance_algo.cc
M src/kudu/rebalance/rebalance_algo.h
M src/kudu/rebalance/rebalancer.cc
M src/kudu/rebalance/rebalancer.h
M src/kudu/tools/ksck-test.cc
M src/kudu/tools/ksck.cc
M src/kudu/tools/ksck.h
M src/kudu/tools/ksck_remote.cc
M src/kudu/tools/ksck_results.cc
M src/kudu/tools/rebalancer_tool.cc
M src/kudu/tools/tool.proto
M src/kudu/tools/tool_action_cluster.cc
19 files changed, 1,224 insertions(+), 322 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/18294/2
--
To view, visit http://gerrit.cloudera.org:8080/18294
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7d2e19266e993f5e2ae13ba18d323c83db30eac1
Gerrit-Change-Number: 18294
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
