Shubham Chopra created SPARK-15353:
--------------------------------------
Summary: Making peer selection for block replication pluggable
Key: SPARK-15353
URL: https://issues.apache.org/jira/browse/SPARK-15353
Project: Spark
Issue Type: Sub-task
Components: Block Manager, Spark Core
Reporter: Shubham Chopra
BlockManagers running on executors provide all logistics around block
management. Before a BlockManager can be used, it has to be “initialized”. As a
part of the initialization, BlockManager asks the BlockManagerMasterEndpoint to
give it topology information. The BlockManagerMasterEndpoint is provided a
pluggable interface that can be used to resolve a hostname to topology. This
information is used to decorate the BlockManagerId. This happens at cluster
start and whenever a new executor is added.
During replication, the BlockManager gets the list of all its peers in the form
of a Seq[BlockManagerId]. We add a pluggable prioritizer that can be used to
prioritize this list of peers based on topology information. Peers with higher
priority occur first in the sequence and the BlockManager tries to replicate
blocks in that order.
There would be default implementations for these pluggable interfaces that
replicate the existing behavior of randomly choosing a peer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
