[
https://issues.apache.org/jira/browse/YARN-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054571#comment-14054571
]
Vinod Kumar Vavilapalli commented on YARN-2261:
-----------------------------------------------
The proposal here is to have a YARN application-level cleanup container that
runs only as the last thing for an application in the cluster.
- In a way, we already have this today, as we let AM's hang around for a while
(by default - 10mins) *after* unregister - this feature makes it explicit.
- For those who have lived in this space around for a while, this is akin to
MR job-cleanup.
- This feature lets apps submit a separate container-launch-context for
cleanup, one that is only run after the app is done for real.
- Clearly, it will
-- be optional.
-- Have timeouts on how much time it can take to finish (default,
overridable, and upper limit. Default = today's time for AMs to exit after
unregister?)
-- Have resource requests limits like usual
-- May have its own retries (Cleanup failure != Application failure as
today?)
Some challenges
- Cleanup container may not get resources because cluster may have gotten busy
after the final AM exit. Solution is to reserve (part of) resources used by the
last AM for use by the cleanup container
> YARN should have a way to run post-application cleanup
> ------------------------------------------------------
>
> Key: YARN-2261
> URL: https://issues.apache.org/jira/browse/YARN-2261
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
>
> See MAPREDUCE-5956 for context. Specific options are at
> https://issues.apache.org/jira/browse/MAPREDUCE-5956?focusedCommentId=14054562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14054562.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
