Hello Adar Dembo, Todd Lipcon, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
to look at the new patch set (#2).
Change subject: Allow for reserving disk space for non-Kudu processes
Allow for reserving disk space for non-Kudu processes
Adds gflags to reserve disk space such that Kudu will not use more than
specified. Hadoop calls this functionality "du.reserved".
If a WAL preallocation is attempted while the log disk is past its
reservation limit then a crash will result.
The log block manager will use non-full disks if possible until all of
the disks are full. If a flush or compaction is attempted when all disks
are beyond their configured capacity then the process will crash.
This initial implementation provides a "best effort" approach. Disk
space checks are only done at preallocation time, and if writes continue
beyond the preallocated point (for both a WAL segment and a data block)
those writes will not be prevented. This makes it easier to provide a
"friendly" option where the block manager will divert new writes to
non-full disks, avoiding a hard crash when only one disk is past its
In the future, we may want to add "hard" and "soft" limits, such that
going beyond the soft limit will do what we do today, and going beyond
the hard limit (say, by writing a very large data block past its
preallocation point) will result in a crash.
This patch includes:
* Some unit tests.
* End-to-end test for compaction falling back to non-full disks due to
disk space backpressure and finally crashing when there is no space
left in any data dir.
* End-to-end test for writes failing due to WAL disk space backpressure,
causing a crash.
16 files changed, 596 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/35/3135/2
To view, visit http://gerrit.cloudera.org:8080/3135
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Owner: Mike Percy <mpe...@apache.org>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>