Impala Public Jenkins has submitted this change and it was merged. Change subject: IMPALA-5352: Age out unused file handles from the cache ......................................................................
IMPALA-5352: Age out unused file handles from the cache Currently, a file handle in the file handle cache will only be evicted if the cache reaches its capacity. This means that file handles can be retained for an indefinite amount of time. This is true even for files that have been deleted, replaced, or modified. Since a file handle maintains a file descriptor for local files, this can prevent the disk space from being freed. Additionally, unused file handles are wasted memory. This adds code to evict file handles that have been unused for longer than a specified threshold. A thread periodically checks the file handle cache to see if any file handle should be evicted. The threshold is specified by 'unused_file_handle_timeout_sec'; it defaults to 6 hours. This adds a test to custom_cluster/test_hdfs_fd_caching.py to verify the eviction behavior. Change-Id: Iefe04b3e2e22123ecb8b3e494934c93dfb29682e Reviewed-on: http://gerrit.cloudera.org:8080/7640 Reviewed-by: Matthew Jacobs <[email protected]> Tested-by: Impala Public Jenkins --- M be/src/runtime/disk-io-mgr-handle-cache.h M be/src/runtime/disk-io-mgr-handle-cache.inline.h M be/src/runtime/disk-io-mgr.cc M tests/custom_cluster/test_hdfs_fd_caching.py 4 files changed, 172 insertions(+), 31 deletions(-) Approvals: Impala Public Jenkins: Verified Matthew Jacobs: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/7640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Iefe04b3e2e22123ecb8b3e494934c93dfb29682e Gerrit-PatchSet: 11 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Matthew Jacobs <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
