Hello wangning, Kudu Jenkins, Andrew Wong, Adar Dembo, Hao Hao,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14743
to look at the new patch set (#14).
Change subject: KUDU-3001 Multi-thread to load containers in a data directory
......................................................................
KUDU-3001 Multi-thread to load containers in a data directory
When a data directory has many block containers, a single thread to
load these container files is low efficiency, we can improve it by
multi-threads.
We did some simple benchmarks to verify it. Adjust
'log_container_max_size' to 1GB to generate more containers when do
benchmarks, adjust 'startup_benchmark_data_dir_count_for_testing' to 8
to make sure exsiting concurrent data directories load are effective,
and adjust 'fs_max_thread_count_per_data_dir' and
'startup_benchmark_block_count_for_testing' to different
values, timing 10 times ReopenBlockManager(), in milliseconds,
result details as follow:
disk type: SSD
| new version
Block count old version | 1 thread | 2 threads | 4 threads | 8 threads | 16
threads | 32 threads
100,000 2,375 2,382 2,342 2,372 2,343
2,353 2,393
1,000,000 24,018 23,813 22,628 22,407 22,367
22,636 23,173
2,000,000 50,163 51,120 39,726 37,589 37,671
37,501 37,710
4,000,000 104,051 105,560 90,427 79,778 73,129
73,205 74,947
8,000,000 214,347 216,210 199,456 159,143 157,190
158,798 157,056
disk type: spinning disk
| new version
Block count old version | 1 thread | 2 threads | 4 threads | 8 threads | 16
threads | 32 threads
100,000 3,207 3,347 3,345 3,279 3,237
3,263 3,221
1,000,000 33,659 34,106 32,081 30,261 30,142
30,115 30,876
2,000,000 68,097 74,939 56,976 51,407 50,957
56,299 58,456
4,000,000 146,503 162,389 116,956 104,435 94,905
102,606 100,526
8,000,000 331,201 349,609 267,259 247,069 243,064
247,810 247,472
Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
---
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
4 files changed, 333 insertions(+), 249 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/14743/14
--
To view, visit http://gerrit.cloudera.org:8080/14743
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0721ee4a5a6824db146ba0658e60eec25dd0c65c
Gerrit-Change-Number: 14743
Gerrit-PatchSet: 14
Gerrit-Owner: Yingchun Lai <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Andrew Wong <[email protected]>
Gerrit-Reviewer: Hao Hao <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Yingchun Lai <[email protected]>
Gerrit-Reviewer: wangning <[email protected]>
