[
https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558739#comment-16558739
]
Yongjun Zhang commented on HDFS-8131:
-------------------------------------
HI [~liushaohui],
Thanks much for the nice work here.
I have some comments.
1. This jira is described as "improvement" rather than new feature, it should
be a new feature and be documented.
2. A question related to the question [~Tagar] asked above:
https://issues.apache.org/jira/browse/HDFS-8131?focusedCommentId=15981732&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15981732
Class AvailableSpaceBlockPlacementPolicy extends BlockPlacementPolicyDefault.
But it doesn't change the behavior of choosing the first node in
BlockPlacementPolicyDefault, so even with this new feature, the local DN is
always chosen as the first DN (of course when it is not excluded), and the new
feature only changes the selection of the rest of the two DNs.
3. Wonder if we could have another placement policy that could potentially have
a choice to choose a different DN than local DN for the first node, so we don't
always choose the local DN as the first node.
Would you please share your thoughts?
Thanks.
> Implement a space balanced block placement policy
> -------------------------------------------------
>
> Key: HDFS-8131
> URL: https://issues.apache.org/jira/browse/HDFS-8131
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 3.0.0-alpha1
> Reporter: Liu Shaohui
> Assignee: Liu Shaohui
> Priority: Minor
> Labels: BlockPlacementPolicy
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
> Attachments: HDFS-8131-branch-2.7.patch, HDFS-8131-v1.diff,
> HDFS-8131-v2.diff, HDFS-8131-v3.diff, HDFS-8131.004.patch,
> HDFS-8131.005.patch, HDFS-8131.006.patch, balanced.png
>
>
> The default block placement policy will choose datanodes for new blocks
> randomly, which will result in unbalanced space used percent among datanodes
> after an cluster expansion. The old datanodes always are in high used percent
> of space and new added ones are in low percent.
> Through we can used the external balance tool to balance the space used rate,
> it will cost extra network IO and it's not easy to control the balance speed.
> An easy solution is to implement an balanced block placement policy which
> will choose low used percent datanodes for new blocks with a little high
> possibility. In a not long term, the used percent of datanodes will trend to
> be balanced.
> Suggestions and discussions are welcomed. Thanks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
