Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20717#discussion_r173587545
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -675,6 +676,22 @@ object InferFiltersFromConstraints extends
Rule[LogicalPlan] with PredicateHelpe
}
if (newConditionOpt.isDefined) Join(left, right, joinType,
newConditionOpt) else join
}
+
+ /**
+ * Returns additional constraints which are not enforced on the result
of join operations, but
+ * which can be enforced either on the left or the right side
--- End diff --
ah i see the problem. For left-anti join, although `Join.output` reuse the
attributes from left child output, they are actually different attributes, e.g.
Join may output null values, so we can't generate these constraints in
`Join.validConstraints`.
I think we can override both `allConstraints` and `constraints`, to make
sure these extra constraints appear in `allConstraints`, but not `constraints`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
