Adar Dembo has posted comments on this change. Change subject: block manager: start using the file cache ......................................................................
Patch Set 4: > * Make sure this handles the case where the ulimit is unlimited As we discovered, it doesn't appear possible to set RLIMIT_NOFILE to RLIMIT_INFINITY. To maximize RLIMIT_NOFILE, an admin has to do the following: 1. Maximize /proc/sys/fs/nr_open. On my Ubuntu 16.04 system the max value is 2147483584. 2. Maximize /proc/sys/fs/file-max. On my system the max value can be -1 (i.e. 18446744073709551615). 3. Call setrlimit(RLIMIT_NOFILE, val). My kernel lets me go as high as /proc/sys/fs/nr_open. Given all of this, I think the current behavior is still correct: use half of the available limit. Separately, I've added code to jack up the soft limit to be equal to the hard limit. I've incorporated it into this patch. > * Consider adding a mandatory minimum ulimit with an unsafe escape > hatch flag in order to avoid situations where Kudu thrashes on an > extremely low number of files Okay, though I think I'll rely on benchmark results to determine this minium. > * It would be nice to see a benchmark comparing number of file > descriptors vs. performance on something like YCSB (which is a > workload I would expect would be dependent on accessing many > blocks) Okay. -- To view, visit http://gerrit.cloudera.org:8080/5147 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: Ieeefd31eca340111bc535eac1f982290e7703a88 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Dan Burkert <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-HasComments: No
