IMPALA-3453: S3: Uneven split sizes are generated for Parquet causing execution skew
Previously the Parquet file format was considered by us as a non-splittable file format. However, we have since done some work on our parquet scanner that will assign row groups based on the split that contains them. This allows for us to chop up a parquet file into multiple splits and still have the file be scanned reliably. This patch changes our perception of Parquet as a splittable file format, which now allows synthesizeBlockMetadata() to split a parquet file on S3 into multiple "blocks" instead of assigning one scan range per file, so that there is an even distribution of scan ranges across the cluster, hence minimizing skew greatly. P.S: To control the size of scan ranges for splittable files on S3, you can change the default "block" size for the S3A filesystem which is governed by "fs.s3a.block.size". Its default value is 32MB. Change-Id: Ib1518ad0c89ef35a3b0567c3902e85a41e34bc3d Reviewed-on: http://gerrit.cloudera.org:8080/2968 Reviewed-by: Sailesh Mukil <[email protected]> Tested-by: Internal Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/02d3e937 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/02d3e937 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/02d3e937 Branch: refs/heads/master Commit: 02d3e93774d49b2170079bbe5172d326e03be012 Parents: 1c704f3 Author: Sailesh Mukil <[email protected]> Authored: Thu May 5 14:32:16 2016 -0700 Committer: Tim Armstrong <[email protected]> Committed: Thu May 12 14:18:04 2016 -0700 ---------------------------------------------------------------------- fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/02d3e937/fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java ---------------------------------------------------------------------- diff --git a/fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java b/fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java index 3670aa5..7fa1041 100644 --- a/fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java +++ b/fe/src/main/java/com/cloudera/impala/catalog/HdfsFileFormat.java @@ -225,9 +225,8 @@ public enum HdfsFileFormat { case RC_FILE: case SEQUENCE_FILE: case AVRO: - return true; case PARQUET: - return false; + return true; default: throw new RuntimeException("Unknown HdfsFormat: " + this + " - should never happen!");
