[
https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495060#comment-15495060
]
Eric Payne commented on YARN-4945:
----------------------------------
[~sunilg],
I noticed in the resourcemanager log that the metrics were not as I would
expect after running applications. For example, after 1 application has
completed running, the {{#queue-active-applications}} metrics remains 1 instead
of 0:
{code}
2016-09-16 01:11:10,189 [SchedulerEventDispatcher:Event Processor] INFO
capacity.LeafQueue: Application removed - appId: application_1473988192446_0001
user: hadoop1 queue: glamdring #user-pending-applications: 0
#user-active-applications: 0 #queue-pending-applications: 0
#queue-active-applications: 1
{code}
After 3 applications have run, the metrics are even more unexpected:
{code}
2016-09-16 01:12:34,622 [SchedulerEventDispatcher:Event Processor] INFO
capacity.LeafQueue: Application removed - appId: application_1473988192446_0003
user: hadoop1 queue: glamdring #user-pending-applications: -4
#user-active-applications: 4 #queue-pending-applications: 0
#queue-active-applications: 3
{code}
I believe the cause of this is in {{LeafQueue#getAllApplications}}:
{code}
public Collection<FiCaSchedulerApp> getAllApplications() {
Collection<FiCaSchedulerApp> apps =
pendingOrderingPolicy.getSchedulableEntities();
apps.addAll(orderingPolicy.getSchedulableEntities());
return Collections.unmodifiableCollection(apps);
}
{code}
The call to {{pendingOrderingPolicy.getSchedulableEntities()}} returns the
{{AbstractComparatorOrderingPolicy#schedulableEntities}} object, and then the
call to {{apps.addAll(orderingPolicy.getSchedulableEntities())}} adds
additional {{FiCaSchedulerApp}}'s to {{schedulableEntities}}.
By creating a copy of the return value of
{{pendingOrderingPolicy.getSchedulableEntities()}}, I have been able to verify
that the {{schedulableEntities}} does not have extra entries. For example:
{code}
public Collection<FiCaSchedulerApp> getAllApplications() {
Collection<FiCaSchedulerApp> apps = new TreeSet<FiCaSchedulerApp>(
pendingOrderingPolicy.getSchedulableEntities());
apps.addAll(orderingPolicy.getSchedulableEntities());
return Collections.unmodifiableCollection(apps);
}
{code}
> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
> Key: YARN-4945
> URL: https://issues.apache.org/jira/browse/YARN-4945
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Attachments: Intra-Queue Preemption Use Cases.pdf,
> IntraQueuepreemption-CapacityScheduler (Design).pdf, YARN-2009-wip.2.patch,
> YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch,
> YARN-2009.v1.patch, YARN-2009.v2.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to
> support features like:
> YARN-2009. YARN-2113. YARN-4781.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
