[
https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221591#comment-14221591
]
Bikas Saha commented on YARN-2139:
----------------------------------
Given that this design and possible implementation might go through unstable
rounds and are currently not abstracted enough in the core code, doing this on
a branch seems prudent.
Given that SSDs are becoming common, thinking of storage as only spinning disks
may be limited. Multiple writers may affect each other more negatively on
spinning disk vs SSDs. It may be useful to see if the consideration of storage
could be abstracted into a plugin so that storage could have a different
resource allocation policy by storage type (e.g. allocate/share by spindle for
spinning disk storage vs allocate/share by iops on ssd storage vs
allocate/share by network bandwidth for non-DAS storage). If we can abstract
the policy into a plugin on trunk itself then perhaps we would not need a
branch. Secondly, it will probably take a long time to agree on what a common
policy should be and the consensus decision will probably not be a good fit for
a large percentage of real clusters because of hardware variety. So making this
a plugin would enable quicker development, trial and usage of disk based
allocation compared to arriving at a grand unified allocation model for storage.
> [Umbrella] Support for Disk as a Resource in YARN
> --------------------------------------------------
>
> Key: YARN-2139
> URL: https://issues.apache.org/jira/browse/YARN-2139
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Wei Yan
> Attachments: Disk_IO_Scheduling_Design_1.pdf,
> Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch,
> YARN-2139-prototype.patch
>
>
> YARN should consider disk as another resource for (1) scheduling tasks on
> nodes, (2) isolation at runtime, (3) spindle locality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
