[
https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14492573#comment-14492573
]
Koji Noguchi commented on HDFS-8131:
------------------------------------
Hi Liu. Have you taken a look at [~kihwal]'s HDFS-8041?
In general, recently generated data are more likely to get accessed than the
old ones. In that sense, we don't want all the new blocks to be copied to the
newly added nodes since those can quickly become the bottleneck.
I think HDFS-8041 hits a good balance.
> Implement a space balanced block placement policy
> -------------------------------------------------
>
> Key: HDFS-8131
> URL: https://issues.apache.org/jira/browse/HDFS-8131
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Liu Shaohui
> Assignee: Liu Shaohui
> Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-8131-v1.diff
>
>
> The default block placement policy will choose datanodes for new blocks
> randomly, which will result in unbalanced space used percent among datanodes
> after an cluster expansion. The old datanodes always are in high used percent
> of space and new added ones are in low percent.
> Through we can used the external balance tool to balance the space used rate,
> it will cost extra network IO and it's not easy to control the balance speed.
> An easy solution is to implement an balanced block placement policy which
> will choose low used percent datanodes for new blocks with a little high
> possibility. In a not long term, the used percent of datanodes will trend to
> be balanced.
> Suggestions and discussions are welcomed. Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
