[jira] [Comment Edited] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
[ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631359#comment-16631359 ] Arun Suresh edited comment on YARN-8808 at 9/28/18 5:15 AM: bq. containersUtilization and nodeUtilization in SchedulerNode are always instantiated (to ResourceUtilization.newInstance(0, 0, 0f)), so getNodeUtilization() / getAggregatedContainersUtilization() should never return null, unless I am missing somthing Again, I noticed the NPE in a testcase setup and I had to put the check. It is possible it might not happen in a real cluster setup (and also my branch was a bit stale). Maybe a good Idea to just put the check in there just as a safety. Am +1 on the patch otherwise was (Author: asuresh): bq. containersUtilization and nodeUtilization in SchedulerNode are always instantiated (to ResourceUtilization.newInstance(0, 0, 0f)), so getNodeUtilization() / getAggregatedContainersUtilization() should never return null, unless I am missing somthing Again, I noticed the NPE in a testcase setup and I had to put the check. It is possible it might not happen in a real cluster. > Use aggregate container utilization instead of node utilization to determine > resources available for oversubscription > - > > Key: YARN-8808 > URL: https://issues.apache.org/jira/browse/YARN-8808 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8088-YARN-1011.01.patch, > YARN-8808-YARN-1011.00.patch > > > Resource oversubscription should be bound to the amount of the resources that > can be allocated to containers, hence the allocation threshold should be with > respect to aggregate container utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
[ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624224#comment-16624224 ] Arun Suresh edited comment on YARN-8808 at 9/21/18 9:50 PM: While working on YARN-1013, I check also for whether the aggregateUtilization/nodeUtilization == 0. This implies that nothing is running on the node, which implies that we should not over-allocate on the node right ? Also, I am thinking a combination of containerUtilization + nodeUtilization should be used though. Consider the situation where the container utilization is high but the node utilization is low - Node has capacity for 4 1GB containers, but is currently running 2 containers each using more than 1.9GB - in this case, overallocation should be allowed. was (Author: asuresh): While working on YARN-1013, looks like we should also check for whether the aggregateUtilization/nodeUtilization == 0. This implies that nothing is running on the node, which implies that we should not over-allocate on the node right ? Also, I am thinking a combination of containerUtilization + nodeUtilization should be used though. Consider the situation where the container utilization is high but the node utilization is low - Node has capacity for 4 1GB containers, but is currently running 2 containers each using more than 1.9GB - in this case, overallocation should be allowed. > Use aggregate container utilization instead of node utilization to determine > resources available for oversubscription > - > > Key: YARN-8808 > URL: https://issues.apache.org/jira/browse/YARN-8808 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8088-YARN-1011.01.patch, > YARN-8808-YARN-1011.00.patch > > > Resource oversubscription should be bound to the amount of the resources that > can be allocated to containers, hence the allocation threshold should be with > respect to aggregate container utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
