Github user nsyca commented on the issue:
https://github.com/apache/spark/pull/14661
This code change looks good to me. The logic in
``
filter.constraints.filter(_.isInstanceOf[IsNotNull])
.exists(expr =>
join.left.outputSet.intersect(expr.references).nonEmpty)
``
is useless here. Checking just an ISNOTNULL() predicate that has an
argument from the left table is not sufficient to make the conclusion that the
predicate will filter null rows from the left table. Likewise for the same
logic checking on the right table.
One minor comment: Does filter.constraints subsume filter.condition? By the
reading of the lazy evaluation of the ``val constraints`` in
``QueryPlan.scala`` and the ``override protected def validConstraints`` in
``case class Filter``, I think it does. If so, then the ``++`` in the ``val
conditions`` at line 1331 is redundant. It could just be changed to ``val
conditions = filter.constraints``.
I will not monitor this PR until my return on 8/29.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
