Hello wangning, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14743
to look at the new patch set (#12).
Change subject: KUDU-3001 Multi-thread to load containers in a data directory
......................................................................
KUDU-3001 Multi-thread to load containers in a data directory
When a data directory has many block containers, a single thread to
load these container files is low efficiency, we can improve it by
multi-threads.
We did some simple benchmarks to verify it. Adjust
'log_container_max_size' to 1GB to generate more containers when do
benchmarks, adjust 'startup_benchmark_data_dir_count_for_testing' to 8
to make sure exsiting concurrent data directories load are effective,
and adjust 'fs_thread_count_per_data_dir' and
'startup_benchmark_block_count_for_testing' to different
values, timing 10 times ReopenBlockManager(), result details as follow:
disk type: SSD
| new version
Block count old version | 1 thread | 5 threads
100,000 2,375 2,382 2,359
1,000,000 24,018 23,813 22,525
2,000,000 50,163 51,120 37,302
4,000,000 104,051 105,560 74,339
8,000,000 214,347 216,210 154,795
disk type: spinning disk
| new version
Block count old version | 1 thread | 5 threads
100,000 3,207 3,347 3,272
1,000,000 33,659 34,106 32,512
2,000,000 68,097 74,939 55,194
4,000,000 146,503 162,389 112,068
8,000,000 331,201 349,609 258,011
Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
---
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
4 files changed, 330 insertions(+), 247 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/14743/12
--
To view, visit http://gerrit.cloudera.org:8080/14743
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
Gerrit-Change-Number: 14743
Gerrit-PatchSet: 12
Gerrit-Owner: Yingchun Lai <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <[email protected]>
Gerrit-Reviewer: wangning <[email protected]>
