Hello Dan Burkert, Adar Dembo,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/8007
to review the following change.
Change subject: log_block_manager: switch from google::sparse_hash_map to
sparsepp
......................................................................
log_block_manager: switch from google::sparse_hash_map to sparsepp
sparsepp is updated for C++11 so it enables move semantics for the map
elements. Since the block map uses ref-counted values, being able to move them
is a big win. It also claims to be generally faster even aside from the
ability to support moves.
This improved startup time 7-8x on a real host with ~11M blocks:
Before:
I0907 17:23:50.748055 12507 fs_manager.cc:335] Time spent opening block
manager: real 108.910s user 0.000s sys 0.001s
After:
I0907 17:20:42.277474 10021 fs_manager.cc:335] Time spent opening block
manager: real 14.348s user 0.000s sys 0.001s
The LBM startup benchmark (1M blocks) improved less substantially but still
noticeably:
Before:
I0907 17:16:54.899818 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.612s user 0.035s sys 0.002s
I0907 17:16:57.498205 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.598s user 0.039s sys 0.001s
I0907 17:17:00.100244 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.602s user 0.042s sys 0.000s
I0907 17:17:02.686638 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.586s user 0.042s sys 0.000s
I0907 17:17:05.284050 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.597s user 0.041s sys 0.001s
I0907 17:17:07.884395 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.600s user 0.039s sys 0.001s
I0907 17:17:10.490550 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.606s user 0.040s sys 0.001s
I0907 17:17:13.070114 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.580s user 0.039s sys 0.000s
I0907 17:17:15.667062 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.597s user 0.040s sys 0.001s
I0907 17:17:18.258447 20839 log_block_manager-test.cc:799] Time spent reopening
block manager: real 2.591s user 0.042s sys 0.000s
After:
I0907 17:15:50.645310 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.570s user 0.034s sys 0.001s
I0907 17:15:52.195543 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.550s user 0.037s sys 0.001s
I0907 17:15:53.755209 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.560s user 0.037s sys 0.001s
I0907 17:15:55.263762 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.509s user 0.038s sys 0.001s
I0907 17:15:56.818748 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.555s user 0.037s sys 0.001s
I0907 17:15:58.379680 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.561s user 0.036s sys 0.001s
I0907 17:15:59.913751 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.534s user 0.038s sys 0.000s
I0907 17:16:01.461668 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.548s user 0.037s sys 0.001s
I0907 17:16:03.020823 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.559s user 0.037s sys 0.001s
I0907 17:16:04.549747 20302 log_block_manager-test.cc:799] Time spent reopening
block manager: real 1.529s user 0.035s sys 0.001s
Change-Id: I7397f9cd418782caecf8b2dae2c7bfe2c0e6215c
---
M src/kudu/fs/log_block_manager.h
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
5 files changed, 35 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/07/8007/1
--
To view, visit http://gerrit.cloudera.org:8080/8007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7397f9cd418782caecf8b2dae2c7bfe2c0e6215c
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
