Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19204 )
Change subject: IMPALA-11704: Delay hdfsOpenFile with data cache ...................................................................... IMPALA-11704: Delay hdfsOpenFile with data cache Delays hdfsOpenFile until after data cache lookup if using a data cache. IMPALA-10147 implemented this, but only when using the file handle cache. This patch adds an additional check in case file handle caching is disabled. In networked environments, hdfsOpenFile can take significant time, as observed in a TPC-DS run of q90 where TotalRawHdfsOpenFileTime represented a majority of time spent for HDFS_SCAN_NODE. This patch brings that time to 0 with a primed data cache. Change-Id: I9429a41fb16de27ccb57730203f95559df0dbfb6 Reviewed-on: http://gerrit.cloudera.org:8080/19204 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/io/cache-reader-test-stub.h M be/src/runtime/io/file-reader.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/runtime/io/local-file-reader.cc M be/src/runtime/io/local-file-reader.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M tests/common/skip.py M tests/custom_cluster/test_data_cache.py 10 files changed, 53 insertions(+), 14 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/19204 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9429a41fb16de27ccb57730203f95559df0dbfb6 Gerrit-Change-Number: 19204 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]>
