[
https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623374#comment-13623374
]
Chris Nauroth commented on HDFS-1804:
-------------------------------------
Hi, Aaron. This looks really good. I applied the patch locally and ran the
new tests successfully. Here are a couple of thoughts:
# The algorithm doesn't apply a scaling factor to account for a heterogeneous
set of volumes. For example, consider a mix of 500 GB disks and 2 TB disks,
all initially empty. It would favor the 2 TB disks heavily until the amount of
free space across all volumes (both 500 GB and 2 TB) balanced out. This would
mean that the risk of harming concurrency is higher if using heterogeneous
volumes. Another scaling factor for relative volume capacity could account for
this. I defer to you on whether or not to consider this in scope right now or
skip it and treat it as a potential future feature request to be handled in a
different jira. The scope that I see in the jira description is open-ended. I
don't think you can even get total capacity through {{FsVolumeSpi}} right now,
so I expect it would be a much bigger change.
# Minor thing: the new helper methods in {{GenericTestUtils}} would have
greater symmetry with JUnit's built-in assert methods if the order of arguments
was: expected, actual, delta.
# For initialization of {{balancedPreferencePercent}}:
{code}
private float balancedPreferencePercent = 0.75f;
{code}
Do you want to use
{{DFSConfigKeys#DFS_DATANODE_FSDATASET_VOLUME_CHOOSING_BALANCED_SPACE_PREFERENCE_PERCENT_DEFAULT}}
here? (Though I imagine it won't really matter in practice since it gets
overwritten from config.)
Thanks, Aaron!
> Modify or add a new block-volume device choosing policy that looks at free
> space
> --------------------------------------------------------------------------------
>
> Key: HDFS-1804
> URL: https://issues.apache.org/jira/browse/HDFS-1804
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
> Affects Versions: 2.0.4-alpha
> Reporter: Harsh J
> Assignee: Aaron T. Myers
> Priority: Minor
> Attachments: HDFS-1804.patch, HDFS-1804.patch, HDFS-1804.patch,
> HDFS-1804.patch
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still
> carries the vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free
> space remaining on the disk (or other params) should be a good addition as an
> alternative to vanilla RR.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
