Repository: incubator-impala Updated Branches: refs/heads/master 6d1a130d8 -> 29faca568
IMPALA-4223: Handle truncated file read from HDFS cache While overwriting files on HDFS via Hive it can happen that Impala sees a partially written, cached file. In these cases we did not correctly handle the partial cached read. This change adds a check and triggers a fall back to disk reads for such errors. If the file is partially written to disk, too, then the query will report a file corruption warning through the disk read path. Change-Id: Id1e1fdb0211819c5938956abb13b512350a46f1a Reviewed-on: http://gerrit.cloudera.org:8080/4828 Reviewed-by: Dan Hecht <[email protected]> Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Marcel Kornacker <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/12e34b4c Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/12e34b4c Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/12e34b4c Branch: refs/heads/master Commit: 12e34b4c10e0685a24ad96ddbc82cd7a395474be Parents: 6d1a130 Author: Lars Volker <[email protected]> Authored: Thu Oct 6 15:07:52 2016 +0200 Committer: Marcel Kornacker <[email protected]> Committed: Sun Oct 30 19:35:50 2016 +0000 ---------------------------------------------------------------------- be/src/runtime/disk-io-mgr-scan-range.cc | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/12e34b4c/be/src/runtime/disk-io-mgr-scan-range.cc ---------------------------------------------------------------------- diff --git a/be/src/runtime/disk-io-mgr-scan-range.cc b/be/src/runtime/disk-io-mgr-scan-range.cc index 89092a9..8ebe7a5 100644 --- a/be/src/runtime/disk-io-mgr-scan-range.cc +++ b/be/src/runtime/disk-io-mgr-scan-range.cc @@ -436,13 +436,21 @@ Status DiskIoMgr::ScanRange::ReadFromCache(bool* read_succeeded) { return Status::OK(); } - // Cached read succeeded. + // Cached read returned a buffer, verify we read the correct amount of data. void* buffer = const_cast<void*>(hadoopRzBufferGet(cached_buffer_)); int32_t bytes_read = hadoopRzBufferLength(cached_buffer_); - // For now, entire the entire block is cached or none of it. - // TODO: if HDFS ever changes this, we'll have to handle the case where half - // the block is cached. - DCHECK_EQ(bytes_read, len()); + // A partial read can happen when files are truncated. + // TODO: If HDFS ever supports partially cached blocks, we'll have to distinguish + // between errors and partially cached blocks here. + if (bytes_read < len()) { + stringstream ss; + VLOG_QUERY << "Error reading file from HDFS cache: " << file_ << ". Expected " + << len() << " bytes, but read " << bytes_read << ". Switching to disk read path."; + // Close the scan range. 'read_succeeded' is still false, so the caller will fall back + // to non-cached read of this scan range. + Close(); + return Status::OK(); + } // Create a single buffer desc for the entire scan range and enqueue that. // 'mem_tracker' is NULL because the memory is owned by the HDFS java client,
