GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11649
[SPARK-13811] [SQL] No Push-Down of Constraints-generated Null Filtering of
Compound Expressions
#### What changes were proposed in this pull request?
We generate `IsNotNull` constraint for compound expressions. However, we
should not add extra predicate conditions for these compound expressions. It
will add extra costs for query processing. Normally, it will not filter out any
row but requires extra processing.
For example,
```SQL
testRelation.where('a + 'b === 1)
```
Without this PR, the plan will be like
```
Filter (((a#0 + b#0) = 1) && isnotnull((a#0 + b#0)))
+- LocalRelation [a#0,b#0,c#0]
```
After the PR, constraints-generated null filtering of compound expressions
will not be added.
```
Filter ((a#0 + b#0) = 1))
+- LocalRelation [a#0,b#0,c#0]
```
cc @sameeragarwal @nongli @marmbrus
#### How was this patch tested?
Added a test case for Filter and another case for Join
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark compoundNullFilter
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11649
----
commit 7b314e1d68960eb47921ad269f47b32b6e1b0fe5
Author: gatorsmile <[email protected]>
Date: 2016-03-11T06:31:22Z
no nulll filtering is added for compound expressions.
commit 9cba8a339a36dba83765954c26905b6d61926d67
Author: gatorsmile <[email protected]>
Date: 2016-03-11T06:31:43Z
Merge remote-tracking branch 'upstream/master' into compoundNullFilter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
