Raju Bairishetti commented on YARN-473:

REST API is showing wrong number of pending applications as it was populating 
the value from *pendingApplications set* (i.e. size of pendingApplications 
set). IMO, it should get the value from Queue metrics instead of getting from 
in-memory data structures.

LeafQueue : 
1) *pendingApplications* : all submitted applications will be added to this set 

2) *activeApplications* :  applications added to this set from 
pendingApplications set if the number of active applications are lesser than 
maximum number of active applications. 

Jmx metrics showing appsPending metric properly. AppsPending will be 
incremented when the application is submitted to Queue and decremented only 
when the application is actually launched(i.e. allocated some resources to it). 

IMO, Rest call for queue info should also use queue metrics instead of 
depending on other data structures. Applications from pendingApplications set 
will be removed before the launching of application.

pendingApplciations set contains the applications which are not ready for 
schedulable at this moment.

One more interesting fact, CapacitySchedulerPage(in UI) has "Num 
Non-Schedulable Applications" and  getting the value form pendingApplications 
set. Rest API call is correct in this case.

 I am thinking couple of approaches to fix this issue:
1) Rename the pendingApplications to nonSchedulableApplications in 
CapacitySchedulerLeafQueueInfo class and introduce new fields 
(pendingApplications) in the CapacitySchedulerLeafQueueInfo and get this value 
from QueueMetrics.
     refactor activeApplications to schedulableApplications and introducing a 
new field(activeApplications) for the same which tells exactly how many AMs are 

2. Refactor "Num Non-Schedulable Applications:" to "Num of Pending 
Applications"  and  refactor "Num Schedulable Applications" to 
activeApplications. Update properly through QueueMetrics instead of depending 
on in-memory data structures in the LeafQueue to get the values.

> Capacity Scheduler webpage and REST API not showing correct number of pending 
> applications
> ------------------------------------------------------------------------------------------
>                 Key: YARN-473
>                 URL: https://issues.apache.org/jira/browse/YARN-473
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 0.23.6
>            Reporter: Kendall Thrapp
>            Assignee: Mit Desai
>              Labels: usability
> The Capacity Scheduler REST API 
> (http://hadoop.apache.org/docs/r0.23.6/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API)
>  is not returning the correct number of pending applications.  
> numPendingApplications is almost always zero, even if there are dozens of 
> pending apps.
> In investigating this, I discovered that the Resource Manager's Scheduler 
> webpage is also showing an incorrect but different number of pending 
> applications.  For example, the cluster I'm looking at right now currently 
> has 15 applications in the ACCEPTED state, but the Cluster Metrics table near 
> the top of the page says there are only 2 pending apps.  The REST API says 
> there are zero pending apps.

This message was sent by Atlassian JIRA

Reply via email to