Wei Yan updated YARN-2139:
    Attachment: YARN-2139-prototype.patch

I submit a prototype of the code implementation, to illustrate the basic design 
and implementation.
Code changes in three major parts:
(1) API: add vdisks as a 3rd type of resources, besides CPU/memory. The NM will 
specifly its own vdisks resource, and the AM includes vdisks in the resource 
(2) Scheduler: the scheduler will consider vdisks availability when scheduling. 
Additionally, the DRF policy also considers vdisks when choosing the dominant 
(3) I/O isolation: this is implemented in the NM side. Use cgroup's blkio 
system to do the container I/O isolation.

Will separate the patch into several sub-task patches once collecting more 
comments and the design, implementation.

> Add support for disk IO isolation/scheduling for containers
> -----------------------------------------------------------
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>            Assignee: Wei Yan
>         Attachments: Disk_IO_Scheduling_Design_1.pdf, 
> Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype.patch
> YARN should support considering disk for scheduling tasks on nodes, and 
> provide isolation for these allocations at runtime.

This message was sent by Atlassian JIRA

Reply via email to