[
https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-2139:
-----------------------------------
Attachment: Disk_IO_Isolation_Scheduling_3.pdf
A bunch of us (Jian, Karthik, Ram, Vinod, Wei) met up offline last week. Vinod
suggested we follow a three-phase approach - (1) avoid overallocation of disk
resources, (2) NMs to provide isolation to disallow a container messing with
another, (3) add scheduling support for diverse disk IO requirements between
containers. We expect little disagreement on the first two items, and
understand consensus on the third item might take a little time. So, we want to
proceed with getting the patches in for (1) and (2) while we arbitrate on (3).
The other primary concern in the discussion and other JIRA comments is the
ability to view disk resource in other dimensions - bandwidth, iops etc.
I have updated the design document accordingly:
# Added a section on the Scope of the work
# Updated the approach to reflect the three main phases involved
# Updated design to plug-in isolation and scheduling along different dimensions
The development would be on a branch - YARN-2139, but would like the patches to
go through the same review-commit process as they would on trunk.
> [Umbrella] Support for Disk as a Resource in YARN
> --------------------------------------------------
>
> Key: YARN-2139
> URL: https://issues.apache.org/jira/browse/YARN-2139
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Wei Yan
> Attachments: Disk_IO_Isolation_Scheduling_3.pdf,
> Disk_IO_Scheduling_Design_1.pdf, Disk_IO_Scheduling_Design_2.pdf,
> YARN-2139-prototype-2.patch, YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on
> nodes, (2) isolation at runtime, (3) spindle locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
