Wei Yan commented on YARN-2139:

Thanks for the comments, [~ste...@apache.org].
For you mentioned HDFS read/write problem, we leave it solved by the network 
part, as we also need handle the hdfs replicate traffic. I agree that we should 
avoid confuction with HDFS ''fs''.

The idea of vdisks follows the vcores, where each physical cpu core is measured 
as some number of vcores. One concern about using real number is that users 
cannot specify their task requirements easily. One way may solve that is to 
provide several levels (low, moderate, high, etc) instead of real numbers. This 
is also similar to the discussions YARN-1024 on how to measure the cpu 
capacity. We can define the how many IOPs/bandwidth map to 1 vdisks.

For the testability, currently I have: (1) For fairshare, start several tasks 
with same operations, put them in a single node, and check their I/O 
performance whether follows fairsharing;  (2) I/O performance isolation for a 
given task, in a fully loaded cluster, we replay the given task several times, 
and verify when its I/O performance is stable. Here the task can do lots of 
local disk read and directly write operation, and the most time is used to do 
the I/O.
Any good testing ideas?

> Add support for disk IO isolation/scheduling for containers
> -----------------------------------------------------------
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>            Assignee: Wei Yan
>         Attachments: Disk_IO_Scheduling_Design_1.pdf

This message was sent by Atlassian JIRA

Reply via email to