[
https://issues.apache.org/jira/browse/SPARK-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524469#comment-14524469
]
Apache Spark commented on SPARK-7317:
-------------------------------------
User 'mridulm' has created a pull request for this issue:
https://github.com/apache/spark/pull/5857
> ShuffleHandle needs to be exposed
> ---------------------------------
>
> Key: SPARK-7317
> URL: https://issues.apache.org/jira/browse/SPARK-7317
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle
> Reporter: Mridul Muralidharan
> Assignee: Mridul Muralidharan
> Priority: Minor
>
> ShuffleHandle is marked private[spark] - while a lot of code which depends on
> it, and exposes it, is DeveloperApi.
> While the actual implementation can remain private[spark], the handle class
> itself should be exposed so that Rdd's can leverage it.
> Example:
> a) ShuffleDependency.shuffleHandle exposes a ShuffleHandle
> b) ShuffleManager instance is exposed via SparkEnv.get.shuffleManager
> c) SparkEnv.get.shuffleManager.getReader is exposed which needs handle as
> param : and can be used to write RDD's which leverage shuffle without needing
> to go through spark's shuffle based rdd's.
> So all the machinery for custom RDD to leverage shuffle exists - except for
> specifying the ShuffleHandle class itself in dependencies.
> This allows for customizations in user code on how to leverage shuffle.
> For example, specialized join implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
