[
https://issues.apache.org/jira/browse/YARN-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582584#comment-16582584
]
Chen Yufei commented on YARN-8513:
----------------------------------
[~leftnoteasy] We encounter the same problem twice today with Hadoop 3.1.0.
ResourceManager log messages are the same as before, flushing out 20k lines
every seconds during deadlock.
Our current setup enables ResourceManager and NameNode HA, killing the active
ResourceManager with SIGKILL will turn other standby RM to active state, and
the new active RM can work properly.
ApplicationMaster running on our cluster is a slightly modified version of
distributedshell (modified from Hadoop 2.7.1). When updating to Hadoop 3.1.0,
we fixed all deprecated APIs and didn't make other modifications. During the RM
deadlock, I killed the AM (having very low CPU usage) which is being reported
in the log message, but RM still flushes the same log messages. If the problem
is caused by our AM, killing the problematic AM may turn RM to a normal state.
But seems like it's the internal deadlock in RM causing the problem.
> CapacityScheduler infinite loop when queue is near fully utilized
> -----------------------------------------------------------------
>
> Key: YARN-8513
> URL: https://issues.apache.org/jira/browse/YARN-8513
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, yarn
> Affects Versions: 3.1.0, 2.9.1
> Environment: Ubuntu 14.04.5 and 16.04.4
> YARN is configured with one label and 5 queues.
> Reporter: Chen Yufei
> Priority: Major
> Attachments: jstack-1.log, jstack-2.log, jstack-3.log, jstack-4.log,
> jstack-5.log, top-during-lock.log, top-when-normal.log
>
>
> ResourceManager does not respond to any request when queue is near fully
> utilized sometimes. Sending SIGTERM won't stop RM, only SIGKILL can. After RM
> restart, it can recover running jobs and start accepting new ones.
>
> Seems like CapacityScheduler is in an infinite loop printing out the
> following log messages (more than 25,000 lines in a second):
>
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue:
> assignedContainer queue=root usedCapacity=0.99816763
> absoluteUsedCapacity=0.99816763 used=<memory:16170624, vCores:1577>
> cluster=<memory:29441544, vCores:5792>}}
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Failed to accept allocation proposal}}
> {{2018-07-10 17:16:29,227 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator:
> assignedContainer application attempt=appattempt_1530619767030_1652_000001
> container=null
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@14420943
> clusterResource=<memory:29441544, vCores:5792> type=NODE_LOCAL
> requestedPartition=}}
>
> I encounter this problem several times after upgrading to YARN 2.9.1, while
> the same configuration works fine under version 2.7.3.
>
> YARN-4477 is an infinite loop bug in FairScheduler, not sure if this is a
> similar problem.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
