Adar Dembo has submitted this change and it was merged. Change subject: KUDU-1508: enforce block limit on lbm containers ......................................................................
KUDU-1508: enforce block limit on lbm containers This patch introduces the following machinery: 1. LBM containers can now be limited to a certain number of blocks using log_container_max_blocks. This is a soft limit; if the limit is reduced and the server restarted, existing containers may exceed it. The default value means "no limit except when vulnerable to KUDU-1508". 2. When we construct a block manager, we check whether it's an el6 kernel (that is, whether it's vulnerable to KUDU-1508). 3. At data directory opening time, we figure out whether it's vulnerable to KUDU-1508. If it is, and if it's running on an ext4 filesystem, we prescribe a particular block limit that depends on the filesystem's block size. These limits were arrived at via experimentation (see commit 4923a74). They are likely more conservative than they need to be (they assume one extent per filesystem block with every other extent punched out), but with the addition of the file cache, the scalability overhead should be minimal. 4. The prescribed limit is passed into each container as it is instantiated. I ran TestContainerWithManyHoles on several loopback ext4 filesystems on el6.6. I tested block sizes of 1024, 2048, and 4096, each with and without the fix. Without the fix, the filesystems needed to be repaired 100% of the time. With the fix, they never needed repair. Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543 Reviewed-on: http://gerrit.cloudera.org:8080/5403 Tested-by: Kudu Jenkins Reviewed-by: Dan Burkert <[email protected]> --- M src/kudu/experiments/KUDU-1508/run_test.sh M src/kudu/fs/block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/util/env.h M src/kudu/util/env_posix.cc 6 files changed, 322 insertions(+), 11 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543 Gerrit-PatchSet: 8 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <[email protected]>
