Josh Rosen created SPARK-8317:
---------------------------------
Summary: Do not push sort into shuffle in Exchange operator
Key: SPARK-8317
URL: https://issues.apache.org/jira/browse/SPARK-8317
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Josh Rosen
Assignee: Josh Rosen
In some cases, Spark SQL pushes sorting operations into the shuffle layer by
specifying a key ordering as part of the shuffle dependency. I think that we
should not do this:
- Since we do not delegate aggregation to Spark's shuffle, specifying the
keyOrdering as part of the shuffle has no effect on the shuffle map side.
- By performing the shuffle ourselves (by inserting a sort operator after the
shuffle instead), we can use the Exchange planner to choose specialized sorting
implementations based on the types of rows being sorted.
- We can remove some complexity from SqlSerializer2 by not requiring it to know
about sort orderings, since SQL's own sort operators will already perform the
necessary defensive copying.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
