[
https://issues.apache.org/jira/browse/HDFS-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096928#comment-15096928
]
Anu Engineer commented on HDFS-9635:
------------------------------------
[~java8964] Thanks for comments. I am looking at this from the perspective of
HDFS-1312. Both HDFS-8538 and HDFS-1312 tries to minimize the internal disk
usage imbalance. That is disks having different amount of data. However if we
*only* use Volume IO as the criteria for selection of where the block will be
placed then we might rapidly run into an issue where a set of small writes all
go to a disk and some large writes to another disk. To avoid that scenario,
would it not make sense to actually combine this with HDFS-1804 and solve
HDFS-8538. Just like placing a block without considering I/O is not very
efficient (as this JIRA illustrates), I think having a block placement policy
without considering how much space is left on volume can create other forms of
inefficiencies. In fact, I think we might have two incomplete solutions,
instead of having a whole working solution. As for the OS level I/O it is an
aspirational goal. It is more than enough if we consider only HDFS I/O
happening to a volume. Would you please let me know if I am missing any use
case for your clusters if we just solve HDFS-8538.
> Add one more volume choosing policy with considering volume IO load
> -------------------------------------------------------------------
>
> Key: HDFS-9635
> URL: https://issues.apache.org/jira/browse/HDFS-9635
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Yong Zhang
> Assignee: Yong Zhang
>
> We have RoundRobinVolumeChoosingPolicy and
> AvailableSpaceVolumeChoosingPolicy, but both not consider volume IO load.
> This jira will add a Add one more volume choosing policy base on how many
> xceiver count on volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
