Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13051 )
Change subject: IMPALA-7957: Fix slot equivalences may be enforced multiple times ...................................................................... IMPALA-7957: Fix slot equivalences may be enforced multiple times Predicates can be divided into three types according to the way they are generated: 1) origin predicates that come from the query 2) auxiliary equal predicates generated for equivalence between a label(alias) and its real expression 3) inferred predicates that inferred from the slot equivalences graph The slot equivalences graph (valueTransferGraph in Analyzer) is generated by the first two kinds of predicates. Analyzer will create equivalence predicates for a PlanNode based on the unassigned predicates and the valueTransferGraph. However, the current implementation can't avoid creating inferred predicates that are duplicated with previously created inferred predicates if they have been assigned before. Duplicated inferred predicates are either redundant or wrong. Say, if we create predicate p1: s1 = s2 for the current PlanNode and p1 duplicates with a previously inferred predicate p0: s1 = s2 (same as s2 = s1), we can prove that p1 is redundant or wrong: 1) p0 must have been assigned. Otherwise, p0 will be in the unassigned conjuncts list and p1 won't be created. 2) p0 must have been assigned to an offspring node of the current PlanNode since we create the PlanNodes in a depth first manner. 3) The origin predicates that infer to p0 have been assigned to an offspring node too. Then, rows that should be rejected have been filtered out either by p0 or the origin predicates that infer to p0. What's worse, assigning p1 on top of the origin predicates may wrongly reject rows. Hence, p1 is either redundant or wrong. In inferring predicates based on slot equivalence (createEquivConjuncts) we should update partialEquivSlots based on the previously assigned equivalence predicates. So slot equivalence won't be enforced multiple times. This patch also adds some useful TRACE level logs. Tests: * Add tests for UNIONs in inline-view.test * Run all tests locally in CORE exploration strategy Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Reviewed-on: http://gerrit.cloudera.org:8080/13051 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test 6 files changed, 482 insertions(+), 13 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13051 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ida2d5d8149b217e18ebae61e136848162503653e Gerrit-Change-Number: 13051 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
