IMPALA-4955: Fix integer overflow in hdfs table size accounting We incorrectly use integer type instead of a long for a variable that tracks the partition data size and that is overflowing.
Testing: Couldn't reproduce it locally but the perf build that was hitting this over TPCDS scale 1000 dataset is green with this fix. Change-Id: I8ee568a72ac038464cfb3e4c225f130770dd8d0f Reviewed-on: http://gerrit.cloudera.org:8080/6133 Reviewed-by: Alex Behm <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/013456d3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/013456d3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/013456d3 Branch: refs/heads/master Commit: 013456d3b462ec2af4e026538848fcad79e98531 Parents: d2c540f Author: Bharath Vissapragada <[email protected]> Authored: Thu Feb 23 15:04:11 2017 -0800 Committer: Impala Public Jenkins <[email protected]> Committed: Fri Feb 24 09:26:25 2017 +0000 ---------------------------------------------------------------------- fe/src/main/java/org/apache/impala/catalog/HdfsTable.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/013456d3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java ---------------------------------------------------------------------- diff --git a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java index 6096ba9..fa6a1b7 100644 --- a/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java +++ b/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java @@ -815,7 +815,7 @@ public class HdfsTable extends Table { // Iterate through the current snapshot of the partition directory listing to // figure out files that were newly added/modified. List<FileDescriptor> newFileDescs = Lists.newArrayList(); - int newPartSizeBytes = 0; + long newPartSizeBytes = 0; for (FileStatus fileStatus : fs.listStatus(partDir)) { if (!FileSystemUtil.isValidDataFile(fileStatus)) continue; String fileName = fileStatus.getPath().getName().toString();
