Todd Lipcon has submitted this change and it was merged. Change subject: log_block_manager: use sparse_hash_map for block map ......................................................................
log_block_manager: use sparse_hash_map for block map Based on my testing, this reduces the memory usage of 1M blocks from ~24M to ~9MB. Lookup performance should be similar. Write performance may be slightly slower but this is relatively infrequent and not on a hot path. I also did an allocation trace of a simple program putting 1M <int,int> pairs into both types of maps. With unordered_map, this resulted in 1M 16-byte allocations and several larger allocations as the hashtable bucket array grew (largest being 8MB). With the sparse_hash_map, the allocation sizes were evenly distributed with ~55K allocations at a bunch of sizes between 16 and 384 bytes, with no large allocations. This should be more efficient in terms of avoiding memory fragmentation or longer resizing pauses as blocks are added. In order for this patch to work on gcc 4.9, I had to patch sparsehash. gcc 4 doesn't support std::is_trivially_copy_constructible. This adds a workaround to the sparsehash source to use boost's implementation of the same. Change-Id: I2e75deba7f7fb8d4f800695f195304df603e4ce9 Reviewed-on: http://gerrit.cloudera.org:8080/6471 Reviewed-by: Adar Dembo <[email protected]> Tested-by: Todd Lipcon <[email protected]> --- M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M thirdparty/download-thirdparty.sh A thirdparty/patches/sparsehash-0001-Add-compatibily-for-gcc-4.x-in-traits.patch 4 files changed, 65 insertions(+), 2 deletions(-) Approvals: Adar Dembo: Looks good to me, approved Todd Lipcon: Verified -- To view, visit http://gerrit.cloudera.org:8080/6471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I2e75deba7f7fb8d4f800695f195304df603e4ce9 Gerrit-PatchSet: 5 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]>
