[
https://issues.apache.org/jira/browse/HADOOP-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515836
]
Hairong Kuang commented on HADOOP-1652:
---------------------------------------
When there is no over-capacity data node in the cluster, a source data node
should be chosen from data nodes whose %used space is above average %used
space, among which we should favor data nodes that are on the same rack as the
target data node. I am not sure if we should favor fuller data nodes or not
because it is more expensive than a random selection, but it makes the
algorithm to converge faster. This raised another question that how we can
guarantee that the algorithm converges if rebalancing is not done in the safe
mode.
> Rebalance data blocks when new data nodes added or data nodes become full
> -------------------------------------------------------------------------
>
> Key: HADOOP-1652
> URL: https://issues.apache.org/jira/browse/HADOOP-1652
> Project: Hadoop
> Issue Type: New Feature
> Components: dfs
> Affects Versions: 0.13.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.15.0
>
>
> When a new data node joins hdfs cluster, it does not hold much data. So any
> map task assigned to the machine most likely does not read local data, thus
> increasing the use of network bandwidth. On the other hand, when some data
> nodes become full, new data blocks are placed on only non-full data nodes,
> thus reducing their read parallelism.
> This jira aims to find an approach to redistribute data blocks when imbalance
> occurs in the cluster. An solution should meet the following requirements:
> 1. It maintains data availablility guranteens in the sense that rebalancing
> does not reduce the number of replicas that a block has or the number of
> racks that the block resides.
> 2. An adminstrator should be able to invoke and interrupt rebalancing from a
> command line.
> 3. Rebalancing should be throttled so that rebalancing does not cause a
> namenode to be too busy to serve any incoming request or saturate the network.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
