Hello Dan Burkert, Todd Lipcon, Kudu Jenkins,

I'd like you to reexamine a change.  Please visit


to look at the new patch set (#5).

Change subject: block manager: start using the file cache

block manager: start using the file cache

This commit integrates the file cache into both the file and log block
managers. The capacity of the cache is determined at runtime by inspecting
the RLIMIT_NOFILE resource limit; because the cache doesn't manage all open
files, we use a rather conservative 50% of the limit. This can be overridden
with a command line flag.

Other changes of note:
- Unlike the FBM, the LBM creates and opens container files for read/write
  in the same operation. Since that kind of behavior isn't supported by the
  file cache, we close the files after creating them, then reopen them
  through the cache. While inelegant, I don't expect this to be problematic.
- block_manager-stress-test now uses a PeriodicOpenFdChecker to make sure
  the file cache is working correctly. Some of the test behavior was tweaked
  to increase the number of blocks.
- BlockManagerTest.CloseManyBlocksTest now uses 1000 blocks, which is enough
  for it to fail with the FBM sans file cache when the process resource
  limit is 1024 open files (the default on my Ubuntu 16.04 installation).
- When constructing a block manager, we first try to increase the
  RLIMIT_NOFILE soft limit for the process to be equal to the hard limit. On
  many systems there's enough of a gap between them that this can add a lot
  of cache capacity "for free".

Change-Id: Ieeefd31eca340111bc535eac1f982290e7703a88
M src/kudu/fs/block_manager-stress-test.cc
M src/kudu/fs/block_manager-test.cc
M src/kudu/fs/block_manager.cc
M src/kudu/fs/block_manager.h
M src/kudu/fs/file_block_manager.cc
M src/kudu/fs/file_block_manager.h
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/rpc/rpc-test.cc
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
M src/kudu/util/pb_util.cc
M src/kudu/util/pb_util.h
13 files changed, 193 insertions(+), 88 deletions(-)

  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/47/5147/5
To view, visit http://gerrit.cloudera.org:8080/5147
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ieeefd31eca340111bc535eac1f982290e7703a88
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Dinesh Bhat <din...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>

Reply via email to