Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/10567#discussion_r53364846
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -858,6 +859,49 @@ object ReorderJoin extends Rule[LogicalPlan] with
PredicateHelper {
}
/**
+ * Elimination of outer joins, if the predicates can restrict the result
sets so that
+ * all null-supplying rows are eliminated
+ *
+ * - full outer -> inner if both sides have such predicates
+ * - left outer -> inner if the right side has such predicates
+ * - right outer -> inner if the left side has such predicates
+ * - full outer -> left outer if only the left side has such predicates
+ * - full outer -> right outer if only the right side has such predicates
+ *
+ * This rule should be executed before pushing down the Filter
+ */
+object OuterJoinElimination extends Rule[LogicalPlan] with PredicateHelper
{
+
+ private def buildNewJoin(filter: Filter, join: Join): Join = {
+
+ val leftHasNonNullPredicate =
filter.constraints.filter(_.isInstanceOf[IsNotNull])
+ .exists(expr =>
join.left.outputSet.intersect(expr.references).nonEmpty)
--- End diff --
With your changes, we can support all the expressions, no matter how
complicated they are!!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
