[
https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15565729#comment-15565729
]
Ming Ma commented on HDFS-10967:
--------------------------------
Thanks [~zhz] for the summary!
* Good idea to merge AvailableSpaceBlockPlacementPolicy to
BlockPlacementPolicyDefault as configuration. That will make it easier for
people to experiment the feature without changing the block placement policy.
Generally what we want is the ability to reuse or compose new policies based on
existing block placement policies, as I mentioned in
https://issues.apache.org/jira/browse/HDFS-7613?focusedCommentId=14629070&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14629070.
* I agree with what [~shv] that the admin and ClientProtocol changes seem
unnecessary. We don't define specific commands for many other things, for
example, changing the block placement policy. NN failover in the HA setting
should be good enough.
> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> -----------------------------------------------------------------------
>
> Key: HDFS-10967
> URL: https://issues.apache.org/jira/browse/HDFS-10967
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Labels: balancer
> Attachments: HDFS-10967.00.patch, HDFS-10967.01.patch,
> HDFS-10967.02.patch, HDFS-10967.03.patch
>
>
> Large production clusters are likely to have heterogeneous nodes in terms of
> storage capacity, memory, and CPU cores. It is not always possible to
> proportionally ingest data into DataNodes based on their remaining storage
> capacity. Therefore it's possible for a subset of DataNodes to be much closer
> to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of
> low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very
> useful if we can lower the chance for those near-full DataNodes to become
> destinations for the 2nd and 3rd replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
