Hello Kudu Jenkins, Abhishek Chennaka,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18946
to look at the new patch set (#5).
Change subject: [Tools] Relance data in different disks according to container
size
......................................................................
[Tools] Relance data in different disks according to container size
There already existed a good algorithm to make data distribute in different
disks balancely. But when a host adding more new disks, Kudu can not
rebalance the data in disks immediately.
Therefore, this patch rebalances the data in disks offline. It will statistics
the disk consumption of containers of different disks. Rebalance the data
according
to the file size of containers. Only log-block type is supported.
The algorithm is simple. Sort disks accroding to the size of all log block
containers.
Travel the sorted disk, try to move a log block container from highest
consumption disk
to lowest one, util the highest is less or equal than the average size, or the
least is
more or equal than the average size, or can not find a suitable log block
container
to be moved. This algorithm can not achieve totally balance once.
Why not take tablet as the minimum uint of moving data? A tablet may be
distributed in
different disks controled by parameter FLAGS_fs_target_data_dirs_per_tablet. It
needs to
remap all tablet in different disks. Log block is also unsuitable to taken as
the minimun
unit. Log block is not existed as a unique file. Moving a log block is
difficult. It needs
to update all metadata in superblock. Log block container has no status and it
only contains
two file: .data file and .metadata file. It is easy to achieve file moving.
Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
---
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_fs.cc
4 files changed, 263 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/18946/5
--
To view, visit http://gerrit.cloudera.org:8080/18946
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I368d718e032bec9516cb165ab65c7ea3be083ba4
Gerrit-Change-Number: 18946
Gerrit-PatchSet: 5
Gerrit-Owner: Wang Xixu <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
