[
https://issues.apache.org/jira/browse/HDFS-9635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096871#comment-15096871
]
Andrew Wang commented on HDFS-9635:
-----------------------------------
Hmm, so to clarify, do you plan to extend AvailableSpaceVolumeChoosingPolicy
with IO load information, or write a new policy? I'd like to see it in ASVCP if
possible, and if you're already using this policy by default, it sounds like
this would work for you too.
As you mention, IO wait is a great way of measuring load on a disk. We can try
to collect it in HDFS, but the OS also exposes IO wait information (e.g.
iostat). IMO the OS info is better since it's more complete. The OS is aware of
the actual writes to disk, whereas HDFS is getting buffered by page cache.
Also, HDFS's IO wait info will only be as up-to-date as the last time it wrote,
which is an issue when HDFS shares disks with other apps like MR (common).
In any case, I'm sure there'll be some experimentation to find the right
signals and thresholds. Looking forward to your findings!
> Add one more volume choosing policy with considering volume IO load
> -------------------------------------------------------------------
>
> Key: HDFS-9635
> URL: https://issues.apache.org/jira/browse/HDFS-9635
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: Yong Zhang
> Assignee: Yong Zhang
>
> We have RoundRobinVolumeChoosingPolicy and
> AvailableSpaceVolumeChoosingPolicy, but both not consider volume IO load.
> This jira will add a Add one more volume choosing policy base on how many
> xceiver count on volume.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
