GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2692
[SPARK-3831] Filter rule Improvement and bool expression optimization.
If we write the filter which is always FALSE like
SELECT * from person WHERE FALSE;
200 tasks will run. I think, 1 task is enough.
And current optimizer cannot optimize the case NOT is duplicated like
SELECT * from person WHERE NOT ( NOT (age > 30));
The filter rule above should be simplified
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark SPARK-3831
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2692.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2692
----
commit 8ea872b0131f75ae0797add9bda6dbc79d92736a
Author: Kousuke Saruta <[email protected]>
Date: 2014-10-07T12:34:06Z
Fixed the number of tasks when the data of LocalRelation is empty.
Added optimization rule related to bool expression.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
