[ https://issues.apache.org/jira/browse/YARN-10192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057259#comment-17057259 ]
Jonathan Hung commented on YARN-10192: -------------------------------------- cc [~ebadger], [~epayne], [~leftnoteasy] This was the issue I mentioned during the community call. > CapacityScheduler stuck in loop rejecting allocation proposals > -------------------------------------------------------------- > > Key: YARN-10192 > URL: https://issues.apache.org/jira/browse/YARN-10192 > Project: Hadoop YARN > Issue Type: Bug > Affects Versions: 2.10.0 > Reporter: Jonathan Hung > Priority: Major > > On a 2.10.0 cluster, we observed containers were being scheduled very slowly. > Based on logs, it seems to reject a bunch of allocation proposals, then > accept a bunch of reserved containers, but very few containers are actually > getting allocated: > {noformat} > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,965 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,968 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,968 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,968 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,968 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,977 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,977 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,977 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,977 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,981 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,982 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,982 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,982 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,985 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,985 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,985 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,985 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,988 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,988 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,989 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,989 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:48,995 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:48,995 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,995 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:48,995 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:49,074 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:49,074 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,074 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,074 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:49,084 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8957.grid.linkedin.com:8041 > 2020-03-10 06:31:49,084 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,084 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,095 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15460 > resource=<memory:2048, vCores:1> queue=misc_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,096 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,096 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,096 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:49,104 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > assignedContainer application attempt=appattempt_1582403122262_15460_000001 > container=null queue=misc_default clusterResource=<memory:132329472, > vCores:34413, yarn.io/gpu: 1241> type=OFF_SWITCH requestedPartition=cpu > 2020-03-10 06:31:49,104 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=misc usedCapacity=0.0031771248 > absoluteUsedCapacity=3.1771246E-4 used=<memory:17408, vCores:4> > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,104 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: > assignedContainer queue=root usedCapacity=0.30113637 > absoluteUsedCapacity=0.30113637 used=<memory:16430080, vCores:3343, > yarn.io/gpu: 265> cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,104 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Failed to accept allocation proposal > 2020-03-10 06:31:49,132 INFO > com.linkedin.hadoop.yarn.ams.NodeLabelRoutingAMSProcessor: Applying node > label cpu to container request from application_1582403122262_15460 > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e43_1582403122262_15460_01_000283 Container Transitioned from > ACQUIRED to RELEASED > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest > IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1582403122262_15460 > CONTAINERID=container_e43_1582403122262_15460_01_000283 > RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e43_1582403122262_15460_01_000284 Container Transitioned from > ACQUIRED to RELEASED > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest > IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1582403122262_15460 > CONTAINERID=container_e43_1582403122262_15460_01_000284 > RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e43_1582403122262_15460_01_000285 Container Transitioned from > ACQUIRED to RELEASED > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=azktest > IP=10.150.241.150 OPERATION=AM Released Container TARGET=SchedulerApp > RESULT=SUCCESS APPID=application_1582403122262_15460 > CONTAINERID=container_e43_1582403122262_15460_01_000285 > RESOURCE=<memory:2048, vCores:1> QUEUENAME=misc_default NODELABEL=cpu > 2020-03-10 06:31:49,132 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: > checking for deactivate of application :application_1582403122262_15460 > 2020-03-10 06:31:49,197 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8950.grid.linkedin.com:8041 > 2020-03-10 06:31:49,198 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,198 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,210 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8650.grid.linkedin.com:8041 > 2020-03-10 06:31:49,210 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,210 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,219 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8696.grid.linkedin.com:8041 > 2020-03-10 06:31:49,219 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,219 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,290 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8689.grid.linkedin.com:8041 > 2020-03-10 06:31:49,290 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,290 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,315 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9327.grid.linkedin.com:8041 > 2020-03-10 06:31:49,315 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,315 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,320 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8617.grid.linkedin.com:8041 > 2020-03-10 06:31:49,320 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,320 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9342.grid.linkedin.com:8041 > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8941.grid.linkedin.com:8041 > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,327 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,335 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl3167.grid.linkedin.com:8041 > 2020-03-10 06:31:49,335 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,335 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,357 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8679.grid.linkedin.com:8041 > 2020-03-10 06:31:49,358 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,358 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,360 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl3161.grid.linkedin.com:8041 > 2020-03-10 06:31:49,360 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,360 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,363 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl3197.grid.linkedin.com:8041 > 2020-03-10 06:31:49,363 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,363 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,375 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9364.grid.linkedin.com:8041 > 2020-03-10 06:31:49,375 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,375 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,376 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9367.grid.linkedin.com:8041 > 2020-03-10 06:31:49,376 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,376 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,380 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8668.grid.linkedin.com:8041 > 2020-03-10 06:31:49,380 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,380 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,382 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8952.grid.linkedin.com:8041 > 2020-03-10 06:31:49,383 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,383 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,394 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9329.grid.linkedin.com:8041 > 2020-03-10 06:31:49,394 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,394 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,399 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8682.grid.linkedin.com:8041 > 2020-03-10 06:31:49,399 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,399 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,403 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8708.grid.linkedin.com:8041 > 2020-03-10 06:31:49,404 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,404 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,410 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8932.grid.linkedin.com:8041 > 2020-03-10 06:31:49,410 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,410 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,415 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9340.grid.linkedin.com:8041 > 2020-03-10 06:31:49,415 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,415 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,422 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9378.grid.linkedin.com:8041 > 2020-03-10 06:31:49,422 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,422 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,429 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl3171.grid.linkedin.com:8041 > 2020-03-10 06:31:49,429 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,429 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,430 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl9365.grid.linkedin.com:8041 > 2020-03-10 06:31:49,430 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,430 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,431 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8919.grid.linkedin.com:8041 > 2020-03-10 06:31:49,431 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,431 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,443 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8644.grid.linkedin.com:8041 > 2020-03-10 06:31:49,443 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,443 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,443 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8614.grid.linkedin.com:8041 > 2020-03-10 06:31:49,444 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,444 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,444 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl3176.grid.linkedin.com:8041 > 2020-03-10 06:31:49,444 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,444 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted > 2020-03-10 06:31:49,445 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Trying to fulfill reservation for application > application_1582403122262_15459 on node: ltx1-hcl8615.grid.linkedin.com:8041 > 2020-03-10 06:31:49,445 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator: > Reserved container application=application_1582403122262_15459 > resource=<memory:72704, vCores:2> queue=relevance_default > cluster=<memory:132329472, vCores:34413, yarn.io/gpu: 1241> > 2020-03-10 06:31:49,445 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: > Allocation proposal accepted {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org