[
https://issues.apache.org/jira/browse/YARN-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472246#comment-15472246
]
ASF GitHub Bot commented on YARN-5605:
--------------------------------------
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/124#discussion_r77928672
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
@@ -223,17 +225,76 @@ public void setPolicy(SchedulingPolicy policy)
}
super.policy = policy;
}
-
+
@Override
- public void recomputeShares() {
+ public void updateInternal(boolean checkStarvation) {
readLock.lock();
try {
policy.computeShares(runnableApps, getFairShare());
+ if (checkStarvation) {
+ identifyStarvedApplications();
+ }
} finally {
readLock.unlock();
}
}
+ /**
+ * Helper method to identify starved applications. This needs to be
called
+ * ONLY from {@link #updateInternal}, after the application shares
+ * are updated.
+ *
+ * A queue can be starving due to fairshare or minshare.
+ *
+ * Minshare is defined only on the queue and not the applications.
+ * Fairshare is defined for both the queue and the applications.
+ *
+ * If this queue is starved due to minshare, we need to identify the most
+ * deserving apps if they themselves are not starved due to fairshare.
+ *
+ * If this queue is starving due to fairshare, there must be at least
+ * one application that is starved. And, even if the queue is not
+ * starved due to fairshare, there might still be starved applications.
+ */
+ private void identifyStarvedApplications() {
+ // First identify starved applications and track total amount of
+ // starvation (in resources)
+ Resource fairShareStarvation = Resources.clone(none());
+ TreeSet<FSAppAttempt> appsWithDemand = fetchAppsWithDemand();
+ for (FSAppAttempt app : appsWithDemand) {
+ Resource appStarvation = app.fairShareStarvation();
+ if (Resources.equals(Resources.none(), appStarvation)) {
+ break;
+ } else {
--- End diff --
The else is required. We are iterating through all apps with unmet demand.
Since the list is sorted by fairshare starvation, we could stop iterating when
we hit an app that is at or above its fairshare. Added a comment to clarify
that.
> Preempt containers (all on one node) to meet the requirement of starved
> applications
> ------------------------------------------------------------------------------------
>
> Key: YARN-5605
> URL: https://issues.apache.org/jira/browse/YARN-5605
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: yarn-5605-1.patch
>
>
> Required items:
> # Identify starved applications
> # Identify a node that has enough containers from applications over their
> fairshare.
> # Preempt those containers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
