Jonathan Hung created YARN-10192:
------------------------------------
Summary: CapacityScheduler stuck in loop rejecting allocation
proposals
Key: YARN-10192
URL: https://issues.apache.org/jira/browse/YARN-10192
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jonathan Hung
On a 2.10.0 cluster, we observed containers were being scheduled very slowly.
Based on logs, it seems to reject a bunch of allocation proposals, then accept
a bunch of reserved containers, but very few containers are actually getting
allocated:
{noformat}
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,965 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,968 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,977 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,977 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,977 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,977 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,981 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,982 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,982 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,982 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,985 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,985 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,985 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,985 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,988 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,988 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,989 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,989 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:48,995 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:48,995 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,995 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:48,995 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:49,074 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:49,074 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,074 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,074 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:49,084 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8957.grid.linkedin.com:8041
2020-03-10 06:31:49,084 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,084 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,095 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15460
resource=<memory:2048, vCores:1> queue=misc_default cluster=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,096 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:49,104 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1582403122262_15460_000001
container=null queue=misc_default clusterResource=<memory:132329472,
vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu
2020-03-10 06:31:49,104 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=misc usedCapacity=0.0031771248
absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4>
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,104 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.30113637
absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343,
yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,104 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal
2020-03-10 06:31:49,132 INFO
com.linkedin.hadoop.yarn.ams.NodeLabelRoutingAMSProcessor: Applying node label
cpu to container request from application_1582403122262_15460
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_e43_1582403122262_15460_01_000283 Container Transitioned from
ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest
IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp
RESULT=SUCCESS APPID=application_1582403122262_15460
CONTAINERID=container_e43_1582403122262_15460_01_000283 RESOURCE=<memory:2048,
vCores:1> QUEUENAME=misc_default NODELABEL=cpu
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_e43_1582403122262_15460_01_000284 Container Transitioned from
ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest
IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp
RESULT=SUCCESS APPID=application_1582403122262_15460
CONTAINERID=container_e43_1582403122262_15460_01_000284 RESOURCE=<memory:2048,
vCores:1> QUEUENAME=misc_default NODELABEL=cpu
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_e43_1582403122262_15460_01_000285 Container Transitioned from
ACQUIRED to RELEASED
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest
IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp
RESULT=SUCCESS APPID=application_1582403122262_15460
CONTAINERID=container_e43_1582403122262_15460_01_000285 RESOURCE=<memory:2048,
vCores:1> QUEUENAME=misc_default NODELABEL=cpu
2020-03-10 06:31:49,132 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo:
checking for deactivate of application :application_1582403122262_15460
2020-03-10 06:31:49,197 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8950.grid.linkedin.com:8041
2020-03-10 06:31:49,198 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,198 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,210 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8650.grid.linkedin.com:8041
2020-03-10 06:31:49,210 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,210 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,219 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8696.grid.linkedin.com:8041
2020-03-10 06:31:49,219 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,219 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,290 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8689.grid.linkedin.com:8041
2020-03-10 06:31:49,290 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,290 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,315 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9327.grid.linkedin.com:8041
2020-03-10 06:31:49,315 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,315 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,320 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8617.grid.linkedin.com:8041
2020-03-10 06:31:49,320 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,320 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9342.grid.linkedin.com:8041
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8941.grid.linkedin.com:8041
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,327 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,335 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl3167.grid.linkedin.com:8041
2020-03-10 06:31:49,335 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,335 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,357 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8679.grid.linkedin.com:8041
2020-03-10 06:31:49,358 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,358 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,360 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl3161.grid.linkedin.com:8041
2020-03-10 06:31:49,360 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,360 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,363 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl3197.grid.linkedin.com:8041
2020-03-10 06:31:49,363 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,363 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,375 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9364.grid.linkedin.com:8041
2020-03-10 06:31:49,375 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,375 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,376 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9367.grid.linkedin.com:8041
2020-03-10 06:31:49,376 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,376 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,380 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8668.grid.linkedin.com:8041
2020-03-10 06:31:49,380 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,380 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,382 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8952.grid.linkedin.com:8041
2020-03-10 06:31:49,383 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,383 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,394 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9329.grid.linkedin.com:8041
2020-03-10 06:31:49,394 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,394 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,399 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8682.grid.linkedin.com:8041
2020-03-10 06:31:49,399 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,399 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,403 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8708.grid.linkedin.com:8041
2020-03-10 06:31:49,404 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,404 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,410 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8932.grid.linkedin.com:8041
2020-03-10 06:31:49,410 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,410 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,415 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9340.grid.linkedin.com:8041
2020-03-10 06:31:49,415 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,415 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,422 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9378.grid.linkedin.com:8041
2020-03-10 06:31:49,422 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,422 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,429 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl3171.grid.linkedin.com:8041
2020-03-10 06:31:49,429 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,429 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,430 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl9365.grid.linkedin.com:8041
2020-03-10 06:31:49,430 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,430 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,431 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8919.grid.linkedin.com:8041
2020-03-10 06:31:49,431 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,431 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,443 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8644.grid.linkedin.com:8041
2020-03-10 06:31:49,443 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,443 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,443 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8614.grid.linkedin.com:8041
2020-03-10 06:31:49,444 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,444 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,444 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl3176.grid.linkedin.com:8041
2020-03-10 06:31:49,444 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,444 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted
2020-03-10 06:31:49,445 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Trying to fulfill reservation for application application_1582403122262_15459
on node: ltx1-hcl8615.grid.linkedin.com:8041
2020-03-10 06:31:49,445 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
Reserved container application=application_1582403122262_15459
resource=<memory:72704, vCores:2> queue=relevance_default
cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241>
2020-03-10 06:31:49,445 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Allocation proposal accepted {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
