GitHub user sameeragarwal opened a pull request:
https://github.com/apache/spark/pull/11511
[SPARK-13668][SQL] Reorder filter/join predicates to short-circuit
isNotNull checks
## What changes were proposed in this pull request?
If a filter predicate or a join condition consists of `IsNotNull` checks,
we should reorder these checks such that these non-nullability checks are
evaluated before the rest of the predicates.
For e.g., if a filter predicate is of the form `a > 5 && isNotNull(b)`, we
should rewrite this as `isNotNull(b) && a > 5` during physical plan generation.
## How was this patch tested?
new unit tests that verify the physical plan for both filters and joins in
`ReorderedPredicateSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sameeragarwal/spark reorder-isnotnull
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11511.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11511
----
commit 9341da6bc45c868d14c4f5d4c020e40a6b5ba593
Author: Sameer Agarwal <[email protected]>
Date: 2016-03-02T23:57:57Z
Reorder conditions in join and filters
commit dfb33ecd27bb65903dd4a0a2cd6bfcd0d8d912c3
Author: Sameer Agarwal <[email protected]>
Date: 2016-03-03T01:20:18Z
unit tests: ReorderedPredicateSuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
