Josh Rosen created SPARK-10704:
----------------------------------
Summary: Consolidate HashShuffleReader and ShuffleReader and
refactor ShuffleManager.getReader()
Key: SPARK-10704
URL: https://issues.apache.org/jira/browse/SPARK-10704
Project: Spark
Issue Type: Sub-task
Components: Shuffle
Reporter: Josh Rosen
Assignee: Josh Rosen
The current shuffle code has an interface named ShuffleReader with only one
implementation, HashShuffleReader. This naming is confusing, since the same
read path code is used for both sort- and hash-based shuffle. We should
consolidate these classes.
In addition, there are aspects of ShuffleManager.getReader()'s API which don't
make a lot of sense: it exposes the ability to request a contiguous range of
shuffle partitions, but this feature isn't supported by any ShuffleReader
implementations and isn't used anywhere in the existing code. We should clean
this up, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
