Yingchun Lai has uploaded this change for review. (
http://gerrit.cloudera.org:8080/14743
Change subject: KUDU-3001 Multi-thread to load containers in a data directory
......................................................................
KUDU-3001 Multi-thread to load containers in a data directory
When a data directory has many block containers, a single thread to
load these container files is low efficiency, we can improve it by
multi-threads.
We did some simple benchmarks to verify it. Adjust
'log_container_max_size' to 1GB to generate more containers when do
benchmarks, adjust 'startup_benchmark_data_dir_count_for_testing' to 8
to make sure exsiting concurrent data directories load are effective,
and adjust 'startup_benchmark_block_count_for_testing' to different
values, timing 10 times ReopenBlockManager(), result details as follow:
Block count single thread multi-threads
100,000 2,375 2,320
1,000,000 24,018 22,564
2,000,000 50,163 37,447
4,000,000 104,051 74,666
Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
---
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/util/trace_metrics.cc
4 files changed, 276 insertions(+), 190 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/14743/1
--
To view, visit http://gerrit.cloudera.org:8080/14743
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
Gerrit-Change-Number: 14743
Gerrit-PatchSet: 1
Gerrit-Owner: Yingchun Lai <[email protected]>
