Guang Yang created YARN-10428:
---------------------------------

             Summary: Zombie applications in the YARN queue using FAIR + 
sizebasedweight
                 Key: YARN-10428
                 URL: https://issues.apache.org/jira/browse/YARN-10428
             Project: Hadoop YARN
          Issue Type: Bug
          Components: capacityscheduler
    Affects Versions: 2.8.5
            Reporter: Guang Yang


Seeing zombie jobs in the YARN queue that uses FAIR and size based weight 
ordering policy .

*Detection:*

The YARN UI shows incorrect number of "Num Schedulable Applications".

*Impact:*

The queue has an upper limit of number of running applications, with zombie 
job, it hits the limit even though the number of running applications is far 
less than the limit. 

*Workaround:*

**Fail-over and restart Resource Manager process.

*Analysis:*

**In the heap dump, we can find the zombie jobs in the `FairOderingPolicy#

schedulableEntities` (see attachment). Take application 
"application_1599157165858_29429" for example, it is still in the  
`FairOderingPolicy#schedulableEntities` set, however, if we check the log of 
resource manager, we can see RM already tried to remove the application:

 

./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 
04:32:19,730 INFO 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue 
(ResourceManager Event Processor): Application removed - appId: 
application_1599157165858_29429 user: svc_di_data_eng queue: core-data 
#user-pending-applications: -3 #user-active-applications: 7 
#queue-pending-applications: 0 #queue-active-applications: 21

 

So it appears RM failed to removed the application from the set.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to