[
https://issues.apache.org/jira/browse/SPARK-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-15353:
------------------------------------
Assignee: Apache Spark
> Making peer selection for block replication pluggable
> -----------------------------------------------------
>
> Key: SPARK-15353
> URL: https://issues.apache.org/jira/browse/SPARK-15353
> Project: Spark
> Issue Type: Sub-task
> Components: Block Manager, Spark Core
> Reporter: Shubham Chopra
> Assignee: Apache Spark
> Attachments: BlockManagerSequenceDiagram.png
>
>
> BlockManagers running on executors provide all logistics around block
> management. Before a BlockManager can be used, it has to be “initialized”. As
> a part of the initialization, BlockManager asks the
> BlockManagerMasterEndpoint to give it topology information. The
> BlockManagerMasterEndpoint is provided a pluggable interface that can be used
> to resolve a hostname to topology. This information is used to decorate the
> BlockManagerId. This happens at cluster start and whenever a new executor is
> added.
> During replication, the BlockManager gets the list of all its peers in the
> form of a Seq[BlockManagerId]. We add a pluggable prioritizer that can be
> used to prioritize this list of peers based on topology information. Peers
> with higher priority occur first in the sequence and the BlockManager tries
> to replicate blocks in that order.
> There would be default implementations for these pluggable interfaces that
> replicate the existing behavior of randomly choosing a peer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
