Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13522 )
Change subject: IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1) ...................................................................... IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1) The file handle cache uses the mtime to distinguish different versions of a file separate. For example, if a file at mtime=1 is overwritten with a version at mtime=2, the old file handle from mtime=1 will not be used for the mtime=2 version. In some codepaths, for legacy reasons, the mtime would be unconditionally set to -1, and this eliminates the ability to distinguish between different versions of files. There is no need to set the mtime to -1. It seems to be a legacy bit of cruft. This removes the mtime=-1 behavior for HDFS scan ranges. It removes mtime from BufferOpts and plumbs the mtime through the scan range codepaths separately. Local non-HDFS files do not use the mtime, so those continue to use mtime=-1. Testing: - Passed core tests Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672 Reviewed-on: http://gerrit.cloudera.org:8080/13522 Reviewed-by: Michael Ho <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-page-index.cc M be/src/exec/scanner-context.cc M be/src/runtime/io/data-cache-test.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/handle-cache.inline.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr.cc 17 files changed, 71 insertions(+), 65 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672 Gerrit-Change-Number: 13522 Gerrit-PatchSet: 7 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Lars Volker <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
