Anubhav Dhoot commented on YARN-3119:

The scheduling should continue like before where we schedule containers. If the 
new container causes the ratio to be exceeded we would kill the offending 
containers. In case we dont exceed the limit, the offending containers could be 
given a chance to succeed which can improve throughput of jobs that have skews 
like this.
If multiple containers are over limit they are all deleted for now. In future 
we can be more sophisticated that we kill containers in reverse order of the 
amount they exceed by or some other criteria, until we go back below the ratio. 
That would be a good second improvement over this. 
In general this jira attempts to make memory a little more of a flexible 

> Memory limit check need not be enforced unless aggregate usage of all 
> containers is near limit
> ----------------------------------------------------------------------------------------------
>                 Key: YARN-3119
>                 URL: https://issues.apache.org/jira/browse/YARN-3119
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>            Reporter: Anubhav Dhoot
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-3119.prelim.patch
> Today we kill any container preemptively even if the total usage of 
> containers for that is well within the limit for YARN. Instead if we enforce 
> memory limit only if the total limit of all containers is close to some 
> configurable ratio of overall memory assigned to containers, we can allow for 
> flexibility in container memory usage without adverse effects. This is 
> similar in principle to how cgroups uses soft_limit_in_bytes.

This message was sent by Atlassian JIRA

Reply via email to