[
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766686#comment-16766686
]
Junping Du commented on YARN-999:
---------------------------------
bq. I am not sure how exactly the reduction of node resources is implemented,
but for the opportunistic containers, you can kill stuff locally at the NMs. So
if you need to free up resources due to resource reduction, you can go over the
opportunistic containers running and kill the long-running ones.
So far, the reduction of node resources won't kill any containers but wait
until container get finished - quite old behavior as no long running service
support when feature get implemented for the first time.
I think we need a generic policy here that can pick up containers to balloon
out resources according to some cost - opportunistic/guaranteed could be one
dimension but could count others - container size, running time, etc.
> In case of long running tasks, reduce node resource should balloon out
> resource quickly by calling preemption API and suspending running task.
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-999
> URL: https://issues.apache.org/jira/browse/YARN-999
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
> Reporter: Junping Du
> Priority: Major
> Attachments: YARN-291.000.patch
>
>
> In current design and implementation, when we decrease resource on node to
> less than resource consumption of current running tasks, tasks can still be
> running until the end. But just no new task get assigned on this node
> (because AvailableResource < 0) until some tasks are finished and
> AvailableResource > 0 again. This is good for most cases but in case of long
> running task, it could be too slow for resource setting to actually work so
> preemption could be hired here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
