Hello Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5146
to look at the new patch set (#11).
Change subject: util: add file cache
......................................................................
util: add file cache
This commit introduces a new kind of cache: the file cache. Its purpose is
to enforce an upper bound on the process' open file descriptors with as
little overhead as possible.
During implementation, I followed these design principles:
1. Reuse the existing LRU cache implementation in cache.{cc,h} so that the
file cache can take advantage of any future improvements there.
2. Preserve existing encapsulation as much as possible. For example, reuse
the various open file interfaces defined in util/env.h instead of blowing
them wide open to get at the underlying fds.
3. Even though the primary use case for the file cache is the log block
manager (to allow for many small containers on el6), the implementation
should be generic enough that it can be used in any block manager, or
anywhere else in a Kudu server process.
In particular, I went back and forth between templated and non-templated
approaches. I settled on using templates once Dan helped me through some of
the thornier issues. This approach drops the capability of opening the same
file multiple "ways" in exchange for far less complexity around deferred
deletion. Going down the template route also means the file cache probably
can't be used outside the block managers.
For more design and implementation notes, see the big block comment in
util/file_cache.h.
Other goodies in this commit:
- A correctness test for the file cache.
- A multi-threaded stress test for the file cache.
- A "checker" (in the style of PeriodicWebUIChecker) for verifying that the
number of open file handles is below what we expect it to be.
- A Cache::Handle deleter, for use with std::unique_ptr.
- A gflag for overriding the number of shards used in ShardedLRUCache, used
to simplify capacity-based tests.
Change-Id: I26d02f71b0a9644de0b669875941adae5f426345
---
M src/kudu/util/CMakeLists.txt
M src/kudu/util/cache.cc
M src/kudu/util/cache.h
A src/kudu/util/file_cache-stress-test.cc
A src/kudu/util/file_cache-test-util.h
A src/kudu/util/file_cache-test.cc
A src/kudu/util/file_cache.cc
A src/kudu/util/file_cache.h
M src/kudu/util/test_util.cc
M src/kudu/util/test_util.h
10 files changed, 1,628 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/5146/11
--
To view, visit http://gerrit.cloudera.org:8080/5146
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I26d02f71b0a9644de0b669875941adae5f426345
Gerrit-PatchSet: 11
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Dinesh Bhat <din...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
