[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-2139: --- Assignee: (was: Wei Yan) > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf, > Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, > YARN-2139-prototype.patch > > > YARN should support considering disk for scheduling tasks on nodes, and > provide isolation for these allocations at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2139: -- Attachment: YARN-2139-prototype-2.patch > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf, > Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype-2.patch, > YARN-2139-prototype.patch > > > YARN should support considering disk for scheduling tasks on nodes, and > provide isolation for these allocations at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2139: -- Attachment: YARN-2139-prototype.patch I submit a prototype of the code implementation, to illustrate the basic design and implementation. Code changes in three major parts: (1) API: add vdisks as a 3rd type of resources, besides CPU/memory. The NM will specifly its own vdisks resource, and the AM includes vdisks in the resource request. (2) Scheduler: the scheduler will consider vdisks availability when scheduling. Additionally, the DRF policy also considers vdisks when choosing the dominant resource. (3) I/O isolation: this is implemented in the NM side. Use cgroup's blkio system to do the container I/O isolation. Will separate the patch into several sub-task patches once collecting more comments and the design, implementation. > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf, > Disk_IO_Scheduling_Design_2.pdf, YARN-2139-prototype.patch > > > YARN should support considering disk for scheduling tasks on nodes, and > provide isolation for these allocations at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-2139: --- Description: YARN should support considering disk for scheduling tasks on nodes, and provide isolation for these allocations at runtime. > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf, > Disk_IO_Scheduling_Design_2.pdf > > > YARN should support considering disk for scheduling tasks on nodes, and > provide isolation for these allocations at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2139: -- Attachment: Disk_IO_Scheduling_Design_2.pdf Update a new design doc including spindle-locality information. Comments are very welcome. I'll create the sub-tasks to upload prelim code for review soon. > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf, > Disk_IO_Scheduling_Design_2.pdf > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2139) Add support for disk IO isolation/scheduling for containers
[ https://issues.apache.org/jira/browse/YARN-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan updated YARN-2139: -- Attachment: Disk_IO_Scheduling_Design_1.pdf Attach a design draft. > Add support for disk IO isolation/scheduling for containers > --- > > Key: YARN-2139 > URL: https://issues.apache.org/jira/browse/YARN-2139 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: Disk_IO_Scheduling_Design_1.pdf > > -- This message was sent by Atlassian JIRA (v6.2#6252)
