[
https://issues.apache.org/jira/browse/HDFS-7742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Roberts updated HDFS-7742:
---------------------------------
Attachment: HDFS-7742-v0.patch
Attached patch. Favors decommissioning nodes a bit by allowing them to go up to
hard limit, otherwise not at all.
> favoring decommissioning node for replication can cause a block to stay
> underreplicated for long periods
> --------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7742
> URL: https://issues.apache.org/jira/browse/HDFS-7742
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.6.0
> Reporter: Nathan Roberts
> Assignee: Nathan Roberts
> Attachments: HDFS-7742-v0.patch
>
>
> When choosing a source node to replicate a block from, a decommissioning node
> is favored. The reason for the favoritism is that decommissioning nodes
> aren't servicing any writes so in-theory they are less loaded.
> However, the same selection algorithm also tries to make sure it doesn't get
> "stuck" on any particular node:
> {noformat}
> // switch to a different node randomly
> // this to prevent from deterministically selecting the same node even
> // if the node failed to replicate the block on previous iterations
> {noformat}
> Unfortunately, the decommissioning check is prior to this randomness so the
> algorithm can get stuck trying to replicate from a decommissioning node.
> We've seen this in practice where a decommissioning datanode was failing to
> replicate a block for many days, when other viable replicas of the block were
> available.
> Given that we limit the number of streams we'll assign to a given node
> (default soft limit of 2, hard limit of 4), It doesn't seem like favoring a
> decommissioning node has significant benefit. i.e. when there is significant
> replication work to do, we'll quickly hit the stream limit of the
> decommissioning nodes and use other nodes in the cluster anyway; when there
> isn't significant replication work then in theory we've got plenty of
> replication bandwidth available so choosing a decommissioning node isn't much
> of a win.
> I see two choices:
> 1) Change the algorithm to still favor decommissioning nodes but with some
> level of randomness that will avoid always selecting the decommissioning node
> 2) Remove the favoritism for decommissioning nodes
> I prefer #2. It simplifies the algorithm, and given the other throttles we
> have in place, I'm not sure there is a significant benefit to selecting
> decommissioning nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
