wangning has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16274
Change subject: [KUDU-1728] parallelize download blocks in tablet-copy-client ...................................................................... [KUDU-1728] parallelize download blocks in tablet-copy-client Parallelize the action of 'Download blocks' in tablet-copy-client. Previsouly download blocks from tablet server is sequential executed, thus actions which strongly relied on 'DownloadBlocks' like recover from other tserver, cluster rebalance are slow. Sometimes use only one thread to download those blocks are slow and bandwidth is not the bottleneck. Here I attached the simple benchmark, the result are averaged by 8 epoch result. The experiment is download all data from remote tserver to local tserver. settings: Remote machine: 8 cores 32g tserver limited to 8g, tserver version 1.10 Local machine: 8 cores 12g memory Tserver has 3 x 500M tablets + 3 x 430M tablets + 3 x 420M tablets and 259 x 8M tablets. All tablets were compacted well with dirskrowset height 1.0. Here the number of tablet-copy-client threadpool and the number of blocks-download threadpool are chosen as variable. tablet-copy-client thread is write as tc. block-download thread is write as bd. Metrics of result is seconds with accuracy of 2 decimal points. time tc-1 tc-2 tc-4 tc-8 bd-1 76.91 48.50 34.27 32.39 bd-2 54.45 43.26 33.27 32.37 bd-4 48.36 37.58 33.31 32.30 bd-8 47.95 38.36 33.98 32.60 Change-Id: Id83abca7a38cf183d9c27d82bb8a022699079e0e --- M src/kudu/tserver/tablet_copy_client.cc M src/kudu/tserver/tablet_copy_client.h 2 files changed, 85 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/74/16274/1 -- To view, visit http://gerrit.cloudera.org:8080/16274 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id83abca7a38cf183d9c27d82bb8a022699079e0e Gerrit-Change-Number: 16274 Gerrit-PatchSet: 1 Gerrit-Owner: wangning <[email protected]>
