Bikas Saha commented on YARN-2139:

Given that this design and possible implementation might go through unstable 
rounds and are currently not abstracted enough in the core code, doing this on 
a branch seems prudent. 
Given that SSDs are becoming common, thinking of storage as only spinning disks 
may be limited. Multiple writers  may affect each other more negatively on 
spinning disk vs SSDs. It may be useful to see if the consideration of storage 
could be abstracted into a plugin so that storage could have a different 
resource allocation policy by storage type (e.g. allocate/share by spindle for 
spinning disk storage vs allocate/share by iops on ssd storage vs 
allocate/share by network bandwidth for non-DAS storage). If we can abstract 
the policy into a plugin on trunk itself then perhaps we would not need a 
branch. Secondly, it will probably take a long time to agree on what a common 
policy should be and the consensus decision will probably not be a good fit for 
a large percentage of real clusters because of hardware variety. So making this 
a plugin would enable quicker development, trial and usage of disk based 
allocation compared to arriving at a grand unified allocation model for storage.

> [Umbrella] Support for Disk as a Resource in YARN 
> --------------------------------------------------
>                 Key: YARN-2139
>                 URL: https://issues.apache.org/jira/browse/YARN-2139
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wei Yan
>         Attachments: Disk_IO_Scheduling_Design_1.pdf, 
> Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, 
> YARN-2139-prototype.patch
> YARN should consider disk as another resource for (1) scheduling tasks on 
> nodes, (2) isolation at runtime, (3) spindle locality. 

This message was sent by Atlassian JIRA

Reply via email to