Hello Dan Burkert, Todd Lipcon,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/5146
to review the following change.
Change subject: util: add file cache
......................................................................
util: add file cache
This commit introduces a new kind of cache: the file cache. Its purpose is
to enforce an upper bound on the process' open file descriptors with as
little overhead as possible.
During implementation, I followed these design principles:
1. Reuse the existing LRU cache implementation in cache.{cc,h} so that the
file cache can take advantage of any future improvements there.
2. Preserve existing encapsulation as much as possible. For example, reuse
the various open file interfaces defined in util/env.h instead of blowing
them wide open to get at the underlying fds.
3. Even though the primary use case for the file cache is the log block
manager (to allow for many small containers on el6), the implementation
should be generic enough that it can be used in any block manager, or
anywhere else in a Kudu server process.
In particular, I went back and forth between templatizing the cache to one
particular open file interface vs. the current approach. The templatized
cache looked like spaghetti due to calls between templatized methods and
generic methods, and it required (surprisingly) more casting as the
underlying LRU cache isn't templatized. Plus, it violated principle #3 in
that it made it impossible to use a single cache for multiple open file
types, and thus for different parts of Kudu.
For more design and implementation notes, see the big block comment in
util/file_cache.h.
Other goodies in this commit:
- A correctness test for the file cache.
- A multi-threaded stress test for the file cache.
- A "checker" (in the style of PeriodicWebUIChecker) for verifying that the
number of open file handles is below what we expect it to be.
- A Cache::Handle deleter, for use with std::unique_ptr.
- A gflag for overriding the number of shareds used in ShardedLRUCache, used
to simplify capacity-based tests.
Change-Id: I26d02f71b0a9644de0b669875941adae5f426345
---
M src/kudu/util/CMakeLists.txt
M src/kudu/util/cache.cc
M src/kudu/util/cache.h
A src/kudu/util/file_cache-stress-test.cc
A src/kudu/util/file_cache-test-util.h
A src/kudu/util/file_cache-test.cc
A src/kudu/util/file_cache.cc
A src/kudu/util/file_cache.h
M src/kudu/util/test_util.cc
M src/kudu/util/test_util.h
10 files changed, 1,716 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/46/5146/1
--
To view, visit http://gerrit.cloudera.org:8080/5146
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I26d02f71b0a9644de0b669875941adae5f426345
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
