[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.013.patch Rebased again, (ver.13) > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.0012.patch, > YARN-5716.002.patch, YARN-5716.003.patch, YARN-5716.004.patch, > YARN-5716.005.patch, YARN-5716.006.patch, YARN-5716.007.patch, > YARN-5716.008.patch, YARN-5716.009.patch, YARN-5716.010.patch, > YARN-5716.011.patch, YARN-5716.013.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.0012.patch Rebased to latest trunk (v12). > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.0012.patch, > YARN-5716.002.patch, YARN-5716.003.patch, YARN-5716.004.patch, > YARN-5716.005.patch, YARN-5716.006.patch, YARN-5716.007.patch, > YARN-5716.008.patch, YARN-5716.009.patch, YARN-5716.010.patch, > YARN-5716.011.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.011.patch > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch, > YARN-5716.009.patch, YARN-5716.010.patch, YARN-5716.011.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.010.patch Attached ver.10 patch rebased to latest trunk, Rewrote tests for YARN-4963. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch, > YARN-5716.009.patch, YARN-5716.010.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5716: - Labels: oct16-hard (was: ) > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch, > YARN-5716.009.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated YARN-5716: - Component/s: capacityscheduler > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Labels: oct16-hard > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch, > YARN-5716.009.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.009.patch Thanks [~sunilg] for comments: For 1), yes it looks like we're doing UL check twice, however, the second time only check one application instead of thousands of apps. So the time spent on UL check for accept/apply is ignoreable from my performance tests. For 2), we will not check allocation-from-reserved container by the outter if (...) check For 3), Jian asked the same question, see my answer https://issues.apache.org/jira/browse/YARN-5716?focusedCommentId=15593292=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15593292: bq. No we cannot do this merge, because it is possible that in the previous reservedContainer != null if, we reserve a new container so the check is not valid. For 4): bq. a reference for resourceRequests is kept and under failure updated back to schedulingInfo object. I do not feel its a clean implementation... Actually we clone the ResourceRequest, and we need to check and make sure ResourceRequest is not changed or still required by the apps. bq. Method is very lengthy. We could take out logic like increase request etc. So it ll be more easier. Done bq. readLock is mostly needed for increased request and for appSchedulingInfo. I think some optimizations here could be done separately in another ticket as improvement. Yeah I agree, I would prefer to keep it as-is unless we find any performance issues. bq. preferring equals instead of if (fromReservedContainer != reservedContainerOnNode) I intended to do that, I want to make sure two reserved container are pointed to the same instance. Please check ver.9 patch > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch, > YARN-5716.009.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.008.patch [~sunilg], make sense. I just uploaded ver.8 patch, which removed all set interfaces from new added APIs. And I also added a comment to ResourceCommitRequest constructor to explain the to-release resource behavior. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch, YARN-5716.008.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.007.patch Attached ver.7 patch, fixed locality statistics and added one more test for that. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch, YARN-5716.007.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.006.patch Rebased to latest trunk (006) > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch, > YARN-5716.006.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.005.patch > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch, YARN-5716.005.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.004.patch Attached ver.4 patch, fixed UT failures and javac warnings. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch, YARN-5716.004.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.003.patch Thanks [~jianhe] for reviews, bq. 1 millisecond ? is this intentional ? This is to avoid busy loop and pinned on CPU, sleep 1 ms can make CPU usage close to 0%. bq. getReservedSchedulerKey vs getAllocatedSchedulerKey: looks like the same ? Yeah they're similiar, since they're not new from this patch, I prefer to address this separately. bq. Too many data structures which wrapps container related info with similar names: CSAssignment, ContainerAllocation, ContainerAllocationContext, SchedulerContainer.. could you consolidate these ? Agree, but I prefer to address them separately, CSAssignment/ContainerAllocation are not introduced by this patch. Relation between ContainerAllocationContext and SchedulerContainer is has-more, so we don't need to consolidate the latter two. bq. SchedulingPlacementSet has a bunch unused methods and introduced unused class ResourceRequestUpdateResult, can we add them when needed ? That is part of the SchedulingPlacementSet definition, I prefer to keep them here. My next patch will start using it. Addressed all other comments. (ver.3) > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch, > YARN-5716.003.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.002.patch Attached ver.2 patch. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch, YARN-5716.002.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Attachment: YARN-5716.001.patch Attached ver.1 patch for review, it is fully functioned, has all changes needed in my mind right now. Will clean up more in the next patch for things like unnecessary comments, indent, etc. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5716.001.patch > > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5716) Add global scheduler interface definition and update CapacityScheduler to use it.
[ https://issues.apache.org/jira/browse/YARN-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5716: - Description: Target of this JIRA: - Definition of interfaces / objects which will be used by global scheduling, this will be shared by different schedulers. - Modify CapacityScheduler to use it. > Add global scheduler interface definition and update CapacityScheduler to use > it. > - > > Key: YARN-5716 > URL: https://issues.apache.org/jira/browse/YARN-5716 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > > Target of this JIRA: > - Definition of interfaces / objects which will be used by global scheduling, > this will be shared by different schedulers. > - Modify CapacityScheduler to use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
