GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/7407
[SPARK-9050] [SQL] Remove unused newOrdering argument from Exchange
(cleanup after SPARK-8317)
SPARK-8317 changed the SQL Exchange operator so that it no longer pushed
sorting into Spark's shuffle layer, a change which allowed more efficient
SQL-specific sorters to be used.
This patch performs some leftover cleanup based on those changes:
- Exchange's constructor should no longer accept a `newOrdering` since it's
no longer used and no longer works as expected.
- `addOperatorsIfNecessary` looked at shuffle input's output ordering to
decide whether to sort, but this is the wrong node to be examining: it needs to
look at whether the post-shuffle node has the right ordering, since shuffling
will not preserve row orderings. Thanks to @davies for spotting this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-9050
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7407.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7407
----
commit 2e467dae8c52754825ccd29619ea3da19a7d2030
Author: Josh Rosen <[email protected]>
Date: 2015-07-14T23:35:46Z
Remove `newOrdering` from Exchange.
commit e86649489158eeb30ce09f10a78663f0a438a4ae
Author: Josh Rosen <[email protected]>
Date: 2015-07-14T23:36:38Z
Refactor addOperatorsIfNecessary to make code clearer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
