[
https://issues.apache.org/jira/browse/YARN-9277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766783#comment-16766783
]
Steven Rand commented on YARN-9277:
-----------------------------------
{code}
+ // We should not preempt container which has been running for a long time.
+ if ((System.currentTimeMillis() - container.getCreationTime()) >=
+ getQueue().getFSContext().getPreemptionConfig()
+ .getToBePreemptedContainerRuntimeThreshold()) {
+ logPreemptContainerPreCheckInfo(
+ "this container already run a long time!");
+ return false;
+ }
+
{code}
I disagree with this because it allows for situations in which starved
applications can't preempt applications that are over their fair shares. If
application A is starved and application B is over its fair share, but happens
to have all its containers running for more than the threshold, then
application A is unable to preempt and will remain starved.
It might be reasonable to sort preemptable containers by runtime and preempt
those that have started most recently. However, I worry that this unfairly
biases the scheduler against applications with shorter-lived tasks.
If code can't be optimized, and really does require very long-running tasks,
then these jobs can be run in a queue from which preemption isn't allowed via
the {{allowPreemptionFrom}} property.
> Add more restrictions In FairScheduler Preemption
> --------------------------------------------------
>
> Key: YARN-9277
> URL: https://issues.apache.org/jira/browse/YARN-9277
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
> Reporter: Zhaohui Xin
> Assignee: Zhaohui Xin
> Priority: Major
> Attachments: YARN-9277.001.patch, YARN-9277.002.patch
>
>
>
> I think we should add more restrictions in fair scheduler preemption.
> * We should not preempt self
> * We should not preempt high priority job
> * We should not preempt container which has been running for a long time.
> * ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
