Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5403
to look at the new patch set (#2).
Change subject: KUDU-1508: enforce block limit on lbm containers
......................................................................
KUDU-1508: enforce block limit on lbm containers
This patch introduces the following machinery:
1. The mount information from /etc/mtab can now be parsed. This doesn't work
for macOS, but it's not vulnerable to KUDU-1508 in the first place.
2. LBM containers can now be limited to a certain number of blocks using
log_container_max_blocks. This is a soft limit; if the limit is reduced
and the server restarted, existing containers may exceed it.
The default value means "no limit except when vulnerable to KUDU-1508".
3. When we construct a block manager, we check whether it's an el6 kernel
(that is, whether it's vulnerable to KUDU-1508).
4. At data directory opening time, we get the mounts and try to find each
data dir's mount, the goal being to figure out what filesystem it's on.
5. When creating a new LBM container, if the underlying data dir is on ext4
and we've got that buggy kernel, we prescribe a particular block limit
depending on the filesystem block size. These limits were arrived at via
experimentation (see commit 4923a74). They are likely more conservative
than they need to be (they assume one extent per filesystem block with
every other extent punched out), but with the addition of the file cache,
the scalability overhead should be minimal.
I ran TestContainerWithManyHoles on several loopback ext4 filesystems on
el6.6. I tested block sizes of 1024, 2048, and 4096, each with and without
the fix. Without the fix, the filesystems needed to be repaired 100% of the
time. With the fix, they never needed repair.
Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543
---
M src/kudu/experiments/KUDU-1508/run_test.sh
M src/kudu/fs/CMakeLists.txt
M src/kudu/fs/block_manager-test.cc
A src/kudu/fs/data_dirs-test.cc
M src/kudu/fs/data_dirs.cc
M src/kudu/fs/data_dirs.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
9 files changed, 505 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/5403/2
--
To view, visit http://gerrit.cloudera.org:8080/5403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <[email protected]>
