Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8945 )
Change subject: IMPALA-6364: Bypass file handle cache for ineligible files ...................................................................... IMPALA-6364: Bypass file handle cache for ineligible files Currently, all HdfsFileHandles are owned and constructed by the file handle cache. When the file handle cache is disabled or the file handle is not eligible for caching, the HdfsFileHandle is stored exclusively in ScanRange::exclusive_hdfs_fh_, but the HdfsFileHandle still comes from the file handle cache. It is created via a call to DiskIoMgr::GetCachedHdfsFileHandle() with 'require_new_handle' set to true and destroyed via DiskIoMgr::ReleaseCachedHdfsFileHandle() with 'destroy_handle' set to true. Recent testing has revealed that the lock on the file handle cache is a bottleneck for workloads with many small remote files. There is no benefit to storing these exclusive file handles in the file handle cache, as they do not participate in the caching. This change introduces DiskIoMgr::GetExclusiveHdfsFileHandle() and DiskIoMgr::ReleaseExclusiveHdfsFileHandle(). These are equivalent to the Get/ReleaseCachedHdfsFileHandle() calls, except they bypass the file handle cache and create/destroy the file handle directly. ScanRange::Open()/Close(), which populates and frees ScanRange::exclusive_hdfs_fh_, now uses these new calls rather than accessing the file handle cache. This avoids the locking entirely, solving the bottleneck. To draw a distinction between the two codepaths, HdfsFileHandle is now an abstract class with two subclasses: - CachedHdfsFileHandles cover all handles that live in file handle cache. Get/ReleaseCachedHdfsFileHandle() use this subclass. - ExclusiveHdfsFileHandles cover all cases where a file handle does not come from the cache. The new Get/ReleaseExclusiveHdfsFileHandle() use this subclass. Separately, testing revealed that increasing the number of partitions for the file handle cache also fixes the contention problem. This changes the file handle cache to make the number of partitions configurable via startup parameter num_file_handle_cache_partitions. This allows mitigation of future bottlenecks without a patch. Change-Id: I4ab52b0884a909a4faeb6692f32d45878ea2838f Reviewed-on: http://gerrit.cloudera.org:8080/8945 Reviewed-by: Joe McDonnell <joemcdonn...@cloudera.com> Tested-by: Impala Public Jenkins --- M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/handle-cache.h M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc 6 files changed, 138 insertions(+), 87 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8945 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4ab52b0884a909a4faeb6692f32d45878ea2838f Gerrit-Change-Number: 8945 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
