[jira] [Commented] (YARN-10192) CapacityScheduler stuck in loop rejecting allocation proposals

2020-03-18 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061976#comment-17061976
 ] 

Jonathan Hung commented on YARN-10192:
--

Thanks. Yeah Tao, agreed, we plan on turning DEBUG on for this class when we 
encounter this again.

[~epayne], we have patches on top of 2.10.0, but not YARN-10009. Looking at 
YARN-10009, seems it could be related. Thanks for the reference.

> CapacityScheduler stuck in loop rejecting allocation proposals
> --
>
> Key: YARN-10192
> URL: https://issues.apache.org/jira/browse/YARN-10192
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Priority: Major
>
> On a 2.10.0 cluster, we observed containers were being scheduled very slowly. 
> Based on logs, it seems to reject a bunch of allocation proposals, then 
> accept a bunch of reserved containers, but very few containers are actually 
> getting allocated:
> {noformat}
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,981 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
>

[jira] [Commented] (YARN-10192) CapacityScheduler stuck in loop rejecting allocation proposals

2020-03-12 Thread Eric Payne (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058049#comment-17058049
 ] 

Eric Payne commented on YARN-10192:
---

[~jhung], I'm not sure this is related, but does the Hadoop 2.10.0 version 
installed on you cluster have any JIRAs on top of it, or is it strictly vanilla 
2.10.0?

Specifically, does it have the fix for YARN-10009?

> CapacityScheduler stuck in loop rejecting allocation proposals
> --
>
> Key: YARN-10192
> URL: https://issues.apache.org/jira/browse/YARN-10192
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Priority: Major
>
> On a 2.10.0 cluster, we observed containers were being scheduled very slowly. 
> Based on logs, it seems to reject a bunch of allocation proposals, then 
> accept a bunch of reserved containers, but very few containers are actually 
> getting allocated:
> {noformat}
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,981 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,982 INFO 
>

[jira] [Commented] (YARN-10192) CapacityScheduler stuck in loop rejecting allocation proposals

2020-03-11 Thread Tao Yang (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057537#comment-17057537
 ] 

Tao Yang commented on YARN-10192:
-

Hi, [~wangda]. 
I'm not sure about this issue, we have found some issues when async-scheduling 
is enabled, this issue seemsnot in the async-scheduling mode according to 
the logs above and it's hard to found the root cause from these logs, I think 
more logs are needed for further analyzing via dynamically updating log level 
of some important classes (such as 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp)
 to DEBUG. BTW, scheduler activities is more useful for debugging but only 
applicable after version-3.3.

> CapacityScheduler stuck in loop rejecting allocation proposals
> --
>
> Key: YARN-10192
> URL: https://issues.apache.org/jira/browse/YARN-10192
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Priority: Major
>
> On a 2.10.0 cluster, we observed containers were being scheduled very slowly. 
> Based on logs, it seems to reject a bunch of allocation proposals, then 
> accept a bunch of reserved containers, but very few containers are actually 
> getting allocated:
> {noformat}
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,981 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default

[jira] [Commented] (YARN-10192) CapacityScheduler stuck in loop rejecting allocation proposals

2020-03-11 Thread Wangda Tan (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057277#comment-17057277
 ] 

Wangda Tan commented on YARN-10192:
---

[~Tao Yang] did you remember to see this issue before?

> CapacityScheduler stuck in loop rejecting allocation proposals
> --
>
> Key: YARN-10192
> URL: https://issues.apache.org/jira/browse/YARN-10192
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Priority: Major
>
> On a 2.10.0 cluster, we observed containers were being scheduled very slowly. 
> Based on logs, it seems to reject a bunch of allocation proposals, then 
> accept a bunch of reserved containers, but very few containers are actually 
> getting allocated:
> {noformat}
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,981 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
>

[jira] [Commented] (YARN-10192) CapacityScheduler stuck in loop rejecting allocation proposals

2020-03-11 Thread Jonathan Hung (Jira) 


[ 
https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057259#comment-17057259
 ] 

Jonathan Hung commented on YARN-10192:
--

cc [~ebadger], [~epayne], [~leftnoteasy]

This was the issue I mentioned during the community call.

> CapacityScheduler stuck in loop rejecting allocation proposals
> --
>
> Key: YARN-10192
> URL: https://issues.apache.org/jira/browse/YARN-10192
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Priority: Major
>
> On a 2.10.0 cluster, we observed containers were being scheduled very slowly. 
> Based on logs, it seems to reject a bunch of allocation proposals, then 
> accept a bunch of reserved containers, but very few containers are actually 
> getting allocated:
> {noformat}
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,965 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root usedCapacity=0.30113637 
> absoluteUsedCapacity=0.30113637 used= yarn.io/gpu: 265> cluster=
> 2020-03-10 06:31:48,977 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
>  Failed to accept allocation proposal
> 2020-03-10 06:31:48,981 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
>  assignedContainer application attempt=appattempt_1582403122262_15460_01 
> container=null queue=misc_default clusterResource= vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=misc usedCapacity=0.0031771248 
> absoluteUsedCapacity=3.1771246E-4 used= 
> cluster=
> 2020-03-10 06:31:48,982 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: 
> assignedContainer queue=root