Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18531 )
Change subject: IMPALA-10453: Support file pruning via runtime filters on Iceberg ...................................................................... Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/18531/4/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java File fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java: http://gerrit.cloudera.org:8080/#/c/18531/4/fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java@268 PS4, Line 268: partField.getTransformType() == TIcebergPartitionTransformType.IDENTITY) > It is because FileMetadtaUtils::CreateTemplateTuple only resolves IDENTITY We cannot create template tuple from the other partition transforms. E.g. if we have a column called 'ts' which is a timestamp, and we have the YEAR() transform on it, then the partition would be e.g. 2022, but in the template tuple we need a timestamp. Maybe we should not evaluate the filters on null slots in the template tuple? But generating runtime filters for other transforms is still useful, because they still efficiently filter out Parquet files based on metadata (stats, dictionary). So it would be good to have tests that check this. -- To view, visit http://gerrit.cloudera.org:8080/18531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7762e1238bdf236b85d2728881a402a2bb41f36a Gerrit-Change-Number: 18531 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 16 Jun 2022 12:00:14 +0000 Gerrit-HasComments: Yes
