Hello Ashwani Raina, Yingchun Lai, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/19479
to look at the new patch set (#4).
Change subject: [KUDU-3447] Limit tablets copying speed
......................................................................
[KUDU-3447] Limit tablets copying speed
Copying tablets from an old cluster to another new cluster is a
high resource consumed operation using the command : kudu
local_replica copy_from_remote. If the data size is very large,
the copying process will last for a long time. Other service
maybe get impacted and become unavailable. Therefore it is
better to limit the tablets copying speed and make the system
more stable. The goal is to balance the tablets copying speed and
the resource consumption.
As copy_from_remote is mainly downloading data from the remote
cluster and writing the data to local file system, it is better to
control the downloading speed to control the resource consumption.
There are some algorithms to implement a rate limiter. This patch
will use the token bucket algorithm implemented by Facebook Folly
library: https://github.com/facebook/folly/blob/main/folly/TokenBucket.h
Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3
---
M CMakeLists.txt
A cmake_modules/FindDoubleConversion.cmake
A cmake_modules/FindFmt.cmake
A cmake_modules/FindFolly.cmake
A cmake_modules/FindLibEvent.cmake
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_action_local_replica.cc
M src/kudu/tserver/tablet_copy_client.cc
M src/kudu/tserver/tablet_copy_client.h
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
13 files changed, 378 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/79/19479/4
--
To view, visit http://gerrit.cloudera.org:8080/19479
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1f4834bfb0718a2b6b1d946975287a11f6be1fe3
Gerrit-Change-Number: 19479
Gerrit-PatchSet: 4
Gerrit-Owner: Wang Xixu <1450306...@qq.com>
Gerrit-Reviewer: Ashwani Raina <ara...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Wang Xixu <1450306...@qq.com>
Gerrit-Reviewer: Yingchun Lai <laiyingc...@apache.org>
