Hello Michael Ho, Lars Volker, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13522
to look at the new patch set (#4).
Change subject: IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1)
......................................................................
IMPALA-8561: Eliminate mtime=-1 for HDFS scan ranges (part 1)
The file handle cache uses the mtime to distinguish
different versions of a file separate. For example,
if a file at mtime=1 is overwritten with a version
at mtime=2, the old file handle from mtime=1 will
not be used for the mtime=2 version.
In some codepaths, for legacy reasons, the mtime
would be unconditionally set to -1, and this
eliminates the ability to distinguish between different
versions of files. There is no need to set the mtime to
-1. It seems to be a legacy bit of cruft.
This removes the mtime=-1 behavior for HDFS scan ranges.
It removes mtime from BufferOpts and plumbs the mtime
through the scan range codepaths separately. Local
non-HDFS files do not use the mtime, so those continue
to use mtime=-1.
Testing:
- Passed core tests
Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672
---
M be/src/exec/base-sequence-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-page-index.cc
M be/src/exec/scanner-context.cc
M be/src/runtime/io/data-cache-test.cc
M be/src/runtime/io/disk-io-mgr-stress.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/tmp-file-mgr.cc
17 files changed, 65 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/13522/4
--
To view, visit http://gerrit.cloudera.org:8080/13522
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I48b7ed60d6ab9104b993237b4fe23de5dc058672
Gerrit-Change-Number: 13522
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Lars Volker <[email protected]>
Gerrit-Reviewer: Michael Ho <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
