[ 
https://issues.apache.org/jira/browse/YARN-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588429#comment-14588429
 ] 

Chris Douglas commented on YARN-3119:
-------------------------------------

Systems that embrace more forgiving resource enforcement are difficult to tune, 
particularly if those jobs run in multiple environments with different 
constraints (as is common when moving from research/test to production). If 
jobs silently and implicitly use more resources than requested, then users only 
learn that their container is under-provisioned when the cluster workload 
shifts, and their pipelines start to fail.

I agree with [~aw]'s 
[feedback|https://issues.apache.org/jira/browse/YARN-3119?focusedCommentId=14303956&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14303956].
 If this workaround is committed, this should be disabled by default and 
strongly discouraged.

> Memory limit check need not be enforced unless aggregate usage of all 
> containers is near limit
> ----------------------------------------------------------------------------------------------
>
>                 Key: YARN-3119
>                 URL: https://issues.apache.org/jira/browse/YARN-3119
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-3119.prelim.patch
>
>
> Today we kill any container preemptively even if the total usage of 
> containers for that is well within the limit for YARN. Instead if we enforce 
> memory limit only if the total limit of all containers is close to some 
> configurable ratio of overall memory assigned to containers, we can allow for 
> flexibility in container memory usage without adverse effects. This is 
> similar in principle to how cgroups uses soft_limit_in_bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to