[
https://issues.apache.org/jira/browse/YARN-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14050539#comment-14050539
]
Karthik Kambatla commented on YARN-2175:
----------------------------------------
I ll let Anubhav provide details on the particular instance we ran into this.
bq. We should try to address the right individual problem with its solution
before we put a band-aid that may still be useful for issues that we cannot
just address directly if any.
Having worked on several MR1 production issues, I see your point. I agree we
should look into and address individual problems. That said, I also believe in
failsafes to avoid bringing down a production cluster or failing a critical job
altogether in the face of hardware issues. That gives us time to fix the
individual issues correctly when we encounter them, instead of hurrying for a
hot fix.
> Container localization has no timeouts and tasks can be stuck there for a
> long time
> -----------------------------------------------------------------------------------
>
> Key: YARN-2175
> URL: https://issues.apache.org/jira/browse/YARN-2175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.4.0
> Reporter: Anubhav Dhoot
> Assignee: Anubhav Dhoot
>
> There are no timeouts that can be used to limit the time taken by various
> container startup operations. Localization for example could take a long time
> and there is no automated way to kill an task if its stuck in these states.
> These may have nothing to do with the task itself and could be an issue
> within the platform.
> Ideally there should be configurable limits for various states within the
> NodeManager to limit various states. The RM does not care about most of these
> and its only between AM and the NM. We can start by making these global
> configurable defaults and in future we can make it fancier by letting AM
> override them in the start container request.
> This jira will be used to limit localization time and we can open others if
> we feel we need to limit other operations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
