[
https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13837238#comment-13837238
]
Junping Du commented on YARN-291:
---------------------------------
> Junping, just saw your comments on YARN-999 . I can help on it.
Thanks! I plan to finish option without timeout in Dec, so it would be great
for you to help on timeout part.
>By different options above, do you mean overCommitTimeoutMills > 0, = 0, <0 ?
>I want to find out more use cases associated with this setting besides
>>graceful decommission. For example, you mentioned preemption for long running
>tasks in YARN-999, is that part of or a different use case from graceful
>>decommission?
Yes. overCommitTimeoutMills value sets different options here. <0 (or just -1)
means we tolerant tasks running to the end even under resource over-consumed
cases; >=0 means we only tolerant a few time specified in
overCommitTimeoutMills. Once timeout, we do aggressive ways (i.e. preemption on
assigned containers with frozen or kill tasks) to reclaim resources so that
NM's resource can get it balanced again. Graceful decommission is just a
special case for this where we always set NM's totalResource to 0 first, so all
assigned containers will get released after a timeout (except timeout = -1). If
we can set a proper timeout value here, then it will get chance for NM to
finish running tasks with intermediate map output get retrieval before
decommissioned and that's why we call it "graceful".
>Also, about the August patch CoreAndAdmin.patch (in YARN-291) , can you let us
>know your plan about it because it seems useful for graceful
> decommission from outside of YARN code.
Most of patches are on the track. YARN-311 (core changes) get checked in,
YARN-312 (RPC) get reviewed with +1. Will be there soon.
Cheers,
> [Umbrella] Dynamic resource configuration
> -----------------------------------------
>
> Key: YARN-291
> URL: https://issues.apache.org/jira/browse/YARN-291
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, scheduler
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments: Elastic Resources for YARN-v0.2.pdf,
> YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
> YARN-291-CoreAndAdmin.patch, YARN-291-JMXInterfaceOnNM-02.patch,
> YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
> YARN-291-YARNClientCommandline-04.patch, YARN-291-all-v1.patch,
> YARN-291-core-HeartBeatAndScheduler-01.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource
> is static during the lifetime of the NM process. Allowing run-time
> configuration on per node resource will give us finer granularity of resource
> elasticity. This allows Hadoop workloads to coexist with other workloads on
> the same hardware efficiently, whether or not the environment is virtualized.
> More background and design details can be found in attached proposal.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
