Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18569
to look at the new patch set (#26).
Change subject: KUDU-3371 [fs] Use RocksDB to store LBM metadata
......................................................................
KUDU-3371 [fs] Use RocksDB to store LBM metadata
Since LogBlockContainer store block records sequentially in
metadata file, the live blocks maybe in a very low ratio,
and it cause disk space wasting and long time bootstrap.
This patch introduce RocksDB to store LBM metadata, a new
item will be Put() into RocksDB when a new block created
in LBM, and the item will be Delete()d from RocksDB when
the block removed from LBM. Data in RocksDB can be
maintained in RocksDB itself, i.e. deleted items will be
GCed so doesn't need rewriting as how we do it in current
LBM.
The implemention also reuse most logic of LBM, the main
difference is store Block records metadata in RocksDB.
1. Make LogBlockManager as a super class
2. The former LBM that stores metadata in a append only
file, is separeted from LBM and specified a new name
LogfBlockManager. Its behavior has no change.
3. Introduce a new class LogrBlockManager that stores
metadata in RocksDB, the main idea:
a. Create container
Data file is created as before, metadata is stored
in keys prefixed by the container's id, append the
block id, e.g. <container_id>.<block_id>. Make sure
there is no such keys in RocksDB before this
container created.
b. Open container
Make sure the data file is healthy.
c. Deconstruct container
If the container is dead (full and no live blocks),
remove the data file, and clean up keys prefixed by
the container's id.
d. Load container (by ProcessRecords())
Iterate the RocksDB in the key range
[<container_id>, <next_container_id>), only live
block records will be populated, we can use them
as before.
e. Create blocks in a container
Put() serialized BlockRecordPB records into RocksDB
in batch, keys are in form of
'<container_id>.<block_id>' as mentioned above.
f. Remove blocks from a container
Contruct the keys by container's id and block's
id, Delete() them from RocksDB in batch.
4. Some refactors, such as create and delete blocks in
batch to reduce lock consult times.
This patch contains the following changes:
- Adds RocksDB as a thirdparty lib
- Adds a new block manager type named 'logr', it use RocksDB
to store LBM metadata, it is also specified by flag
'--block_manager'.
- block_manager-test supports to test LogrBlockManager
- block_manager-stress-test supports to test LogrBlockManager
- log_block_manager-test supports to test LogrBlockManager
- tablet_server-test supports to test LogrBlockManager
- dense_node-itest supports to test LogrBlockManager
- kudu-tool-test supports to test LogrBlockManager
It's optional to use RocksDB, we can use the former LBM as
before, we can introduce more tools to convert data between
the two implemention in the future.
The optimization is obvious as shown in JIRA KUDU-3371, it
shows that reopen staged reduced upto 90% time cost.
Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
---
M CMakeLists.txt
A cmake_modules/FindRocksdb.cmake
M src/kudu/benchmarks/CMakeLists.txt
M src/kudu/client/CMakeLists.txt
M src/kudu/consensus/CMakeLists.txt
M src/kudu/fs/CMakeLists.txt
M src/kudu/fs/block_manager-stress-test.cc
M src/kudu/fs/block_manager-test.cc
M src/kudu/fs/block_manager.h
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/dir_manager.cc
M src/kudu/fs/dir_manager.h
M src/kudu/fs/dir_util.cc
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/fs_manager-test.cc
M src/kudu/fs/fs_manager.cc
M src/kudu/fs/fs_report.cc
M src/kudu/fs/fs_report.h
M src/kudu/fs/log_block_manager-test-util.cc
M src/kudu/fs/log_block_manager-test-util.h
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/integration-tests/CMakeLists.txt
M src/kudu/integration-tests/dense_node-itest.cc
M src/kudu/integration-tests/ts_recovery-itest.cc
M src/kudu/server/CMakeLists.txt
M src/kudu/tablet/compaction-test.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tools/CMakeLists.txt
M src/kudu/tools/kudu-tool-test.cc
M src/kudu/tools/tool_test_util.cc
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/util/CMakeLists.txt
M src/kudu/util/oid_generator-test.cc
M src/kudu/util/oid_generator.cc
M src/kudu/util/oid_generator.h
M thirdparty/build-definitions.sh
M thirdparty/build-thirdparty.sh
M thirdparty/download-thirdparty.sh
M thirdparty/vars.sh
42 files changed, 2,144 insertions(+), 586 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/69/18569/26
--
To view, visit http://gerrit.cloudera.org:8080/18569
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie72f6914eb5653a9c034766c6cd3741a8340711f
Gerrit-Change-Number: 18569
Gerrit-PatchSet: 26
Gerrit-Owner: Yingchun Lai <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
