GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/8823
[SPARK-10539][SQL]Project should not be pushed down through Intersect or
Except #8742
Intersect and Except are both set operators and they use the all the
columns to compare equality between rows. When pushing their Project parent
down, the relations they based on would change, therefore not an equivalent
transformation.
JIRA: https://issues.apache.org/jira/browse/SPARK-10539
I added some comments based on the fix of
https://github.com/apache/spark/pull/8742.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark fix_set_optimization
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8823.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8823
----
commit a9aa64388bae6afffecb27c432d9cf3de77aeba0
Author: Yijie Shen <[email protected]>
Date: 2015-09-14T09:05:38Z
fix set optimization by eliminate empty project push down
commit f9a3b70c7f4c73c7f0572c31a90bb7f7b4698ef7
Author: Yijie Shen <[email protected]>
Date: 2015-09-14T16:30:38Z
Project should not be pushed down through Intersect or Except
commit 1f56d2e626ed75d4ce96ffd70929243eed4c8143
Author: Yin Huai <[email protected]>
Date: 2015-09-18T18:13:05Z
Add comments.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
