[
https://issues.apache.org/jira/browse/YARN-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313097#comment-16313097
]
Jiandan Yang commented on YARN-7693:
-------------------------------------
[~miklos.szeg...@cloudera.com] Opportunistic Containers may impact Guaranteed
Contains when they are under the same group
memory.soft_limit_in_bytes is best-effort and not guaranteed. Just think the
follow steps:
1. memory utilization of Guaranteed Containers in a NodeManager is very low,
real memory usage is under allocation due to little traffic;
2. Scheduler some Opportunistic Containers on that NodeManager due to
oversubscription;
3. Guaranteed Containers memory utilization increases duo to a lot of traffic,
and do not reach the hard limit of them
4. *hadoop-yarn* exceeds hard limit
5. if set oom-killer, Guaranteed Container may be killed, that is not in line
with the principle
6. if not set oom-killer, Guaranteed Container may hang
So Opportunistic Containers may impact Guaranteed Contains when They are under
the same group.
If They are under different groups. Guaranteed and Opportunistic have their own
hard limit, Opportunistic Containers never impact Guaranteed Containers.
Monitor resource utilization of Guaranteed Containers, if there is a gap
between allocation and required, then picking a part of gap resource to
Opportunistic Group; If the gap is less than a given value, then decrease the
hard limit of Guaranteed Group. Kill containers when adjust hard limit fails
for given times in order to make sure the resource of Guaranteed Containers.
> ContainersMonitor support configurable
> --------------------------------------
>
> Key: YARN-7693
> URL: https://issues.apache.org/jira/browse/YARN-7693
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
> Reporter: Jiandan Yang
> Assignee: Jiandan Yang
> Priority: Minor
> Attachments: YARN-7693.001.patch, YARN-7693.002.patch
>
>
> Currently ContainersMonitor has only one default implementation
> ContainersMonitorImpl,
> After introducing Opportunistic Container, ContainersMonitor needs to monitor
> system metrics and even dynamically adjust Opportunistic and Guaranteed
> resources in the cgroup, so another ContainersMonitor may need to be
> implemented.
> The current ContainerManagerImpl ContainersMonitorImpl direct new
> ContainerManagerImpl, so ContainersMonitor need to be configurable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
