Adar Dembo has submitted this change and it was merged. Change subject: block manager: start using the file cache ......................................................................
block manager: start using the file cache This commit integrates the file cache into both the file and log block managers. The capacity of the cache is determined at runtime by inspecting the RLIMIT_NOFILE resource limit; because the cache doesn't manage all open files, we use a rather conservative 50% of the limit. This can be overridden with a command line flag, which can also be used to disable file caching altogether. Other changes of note: - Unlike the FBM, the LBM creates and opens container files for read/write in the same operation. Since that kind of behavior isn't supported by the file cache, we close the files after creating them, then reopen them through the cache. While inelegant, I don't expect this to be problematic. - block_manager-stress-test now uses a PeriodicOpenFdChecker to make sure the file cache is working correctly. Some of the test behavior was tweaked to increase the number of blocks, and to avoid violating the semantics of the file cache. - BlockManagerTest.CloseManyBlocksTest now uses 1000 blocks, which is enough for it to fail with the FBM sans file cache when the process resource limit is 1024 open files (the default on my Ubuntu 16.04 installation). - When constructing a block manager, we first try to increase the RLIMIT_NOFILE soft limit for the process to be equal to the hard limit. On many systems there's enough of a gap between them that this can add a lot of cache capacity "for free". Change-Id: Ieeefd31eca340111bc535eac1f982290e7703a88 Reviewed-on: http://gerrit.cloudera.org:8080/5147 Reviewed-by: Dan Burkert <[email protected]> Tested-by: Kudu Jenkins --- M src/kudu/fs/block_manager-stress-test.cc M src/kudu/fs/block_manager-test.cc M src/kudu/fs/block_manager.cc M src/kudu/fs/block_manager.h M src/kudu/fs/file_block_manager.cc M src/kudu/fs/file_block_manager.h M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/rpc/rpc-test.cc M src/kudu/util/env.h M src/kudu/util/env_posix.cc M src/kudu/util/pb_util.cc M src/kudu/util/pb_util.h 13 files changed, 316 insertions(+), 120 deletions(-) Approvals: Dan Burkert: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ieeefd31eca340111bc535eac1f982290e7703a88 Gerrit-PatchSet: 11 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Dinesh Bhat <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <[email protected]>
