Che Yufei created YARN-8513:
-------------------------------
Summary: CapacityScheduler infinite loop when queue is near fully
utilized
Key: YARN-8513
URL: https://issues.apache.org/jira/browse/YARN-8513
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, yarn
Affects Versions: 2.9.1
Environment: Ubuntu 14.04.5
YARN is configured with one label and 5 queues.
Reporter: Che Yufei
ResourceManager does not respond to any request when queue is near fully
utilized sometimes. Sending SIGTERM won't stop RM, only SIGKILL can. After RM
restart, it can recover running jobs and start accepting new ones.
Seems like CapacityScheduler is in an infinite loop printing out the following
log messages (more than 25,000 lines in a second):
{{2018-07-10 17:16:29,227 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
assignedContainer queue=root usedCapacity=0.99816763
absoluteUsedCapacity=0.99816763 used=<memory:16170624, vCores:1577>
cluster=<memory:29441544, vCores:5792>}}
{{2018-07-10 17:16:29,227 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
Failed to accept allocation proposal}}
{{2018-07-10 17:16:29,227 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
assignedContainer application attempt=appattempt_1530619767030_1652_000001
container=null
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@14420943
clusterResource=<memory:29441544, vCores:5792> type=NODE_LOCAL
requestedPartition=}}
I encounter this problem several times after upgrading to YARN 2.9.1, while the
same configuration works fine under version 2.7.3.
YARN-4477 is an infinite loop bug in FairScheduler, not sure if this is a
similar problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
