Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15371 )
Change subject: IMPALA-9429: Unioned partition columns break partition pruning ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/15371/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15371/1//COMMIT_MSG@14 PS1, Line 14: surrounding code only handles specific cases. Could you also explain why the bug doesn't happen in these two cases: * when there are only one predicate * when predicates are join with AND but not OR I think the answer is, we devided conjuncts by AND op: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/Expr.java#L949 In these cases, the constant predicate is evaluated to false so the union op is marked as hasEmptyResultSet_=true at https://github.com/apache/impala/blob/a007e3caf8d07a7cffbbc96ef099ecd3dc344ce6/fe/src/main/java/org/apache/impala/analysis/Analyzer.java#L1344 However, in the OR case, the whole predicate is not devided and is not marked as constant. So the code path goes to partition pruning and exposes this illegal state. http://gerrit.cloudera.org:8080/#/c/15371/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java: http://gerrit.cloudera.org:8080/#/c/15371/1/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1734 PS1, Line 1734: analyzer.getConstantFolder().rewriteList(opConjuncts, analyzer); I think we don't need to do constant folding each time we substitute predicates. If we just do it here, we just fix the problem in UNION case. The following query still fails on the same IllegalStateException: select year, month from ( select year, 1 month from functional.alltypesagg ) t where year = 2010 or month = 0; Maybe we should add FoldConstantsRule in HdfsPartitionPruner: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java#L89-L90 -- To view, visit http://gerrit.cloudera.org:8080/15371 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c1384c2cd1ad5f7024449196f9a348ecdccb60b Gerrit-Change-Number: 15371 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler <[email protected]> Gerrit-Reviewer: Anurag Mantripragada <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Comment-Date: Mon, 09 Mar 2020 03:37:51 +0000 Gerrit-HasComments: Yes
