[
https://issues.apache.org/jira/browse/SPARK-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953153#comment-15953153
]
Apache Spark commented on SPARK-15352:
--------------------------------------
User 'lins05' has created a pull request for this issue:
https://github.com/apache/spark/pull/17519
> Topology aware block replication
> --------------------------------
>
> Key: SPARK-15352
> URL: https://issues.apache.org/jira/browse/SPARK-15352
> Project: Spark
> Issue Type: New Feature
> Components: Block Manager, Mesos, Spark Core, YARN
> Reporter: Shubham Chopra
> Assignee: Shubham Chopra
>
> With cached RDDs, Spark can be used for online analytics where it is used to
> respond to online queries. But loss of RDD partitions due to node/executor
> failures can cause huge delays in such use cases as the data would have to be
> regenerated.
> Cached RDDs, even when using multiple replicas per block, are not currently
> resilient to node failures when multiple executors are started on the same
> node. Block replication currently chooses a peer at random, and this peer
> could also exist on the same host.
> This effort would add topology aware replication to Spark that can be enabled
> with pluggable strategies. For ease of development/review, this is being
> broken down to three major work-efforts:
> 1. Making peer selection for replication pluggable
> 2. Providing pluggable implementations for providing topology and topology
> aware replication
> 3. Pro-active replenishment of lost blocks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
