[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.6.patch Attached ver.6 addressed [~jianhe]'s comment. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch, YARN-3026.2.patch, YARN-3026.3.patch, YARN-3026.4.patch, YARN-3026.5.patch, YARN-3026.6.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.5.patch Attached ver.5 fixed some offline suggestions from [~jianhe]: - RMContextImpl#setActiveServiceContext, TestCapacityScheduler changes unneeded - FiCaSchedulerApp#assignContainer - assignContainers And one bug fix: - In CS (before this patch), excessReservation won't be completed since ParentQueue.assignContainers doesn't copy the excessReservation field, so CapacityScheduler will not release such excessReservations. - Currently excessReservation and unreserved containers produced by continousReservationLooking are all released at LeafQueue instead of {{CapacityScheduler#allocateContainersToNode}} - Added test for this as well. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch, YARN-3026.2.patch, YARN-3026.3.patch, YARN-3026.4.patch, YARN-3026.5.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.4.patch Attached ver.4 patch, addressed comments from [~jianhe]. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch, YARN-3026.2.patch, YARN-3026.3.patch, YARN-3026.4.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.3.patch Attached ver.3 patch, reworked on the patch, since we have a lot of code changes since ver.2. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch, YARN-3026.2.patch, YARN-3026.3.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.2.patch Rebased against trunk and fixed findbugs warning. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch, YARN-3026.2.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3026) Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp
[ https://issues.apache.org/jira/browse/YARN-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3026: - Attachment: YARN-3026.1.patch Attached ver.1 patch and kick Jenkins. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp --- Key: YARN-3026 URL: https://issues.apache.org/jira/browse/YARN-3026 Project: Hadoop YARN Issue Type: Task Components: capacityscheduler Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3026.1.patch Have a discussion with [~vinodkv] and [~jianhe]: In existing Capacity Scheduler, all allocation logics of and under LeafQueue are located in LeafQueue.java in implementation. To make a cleaner scope of LeafQueue, we'd better move some of them to FiCaSchedulerApp. Ideal scope of LeafQueue should be: when a LeafQueue receives some resources from ParentQueue (like 15% of cluster resource), and it distributes resources to children apps, and it should be agnostic to internal logic of children apps (like delayed-scheduling, etc.). IAW, LeafQueue shouldn't decide how application allocating container from given resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)