[
https://issues.apache.org/jira/browse/YARN-11967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He resolved YARN-11967.
--------------------------------
Fix Version/s: 3.6.0
Hadoop Flags: Reviewed
Resolution: Fixed
> ContainersMonitor skips the polling-based memory check when memory.enforced
> is true even though CGroups memory (yarn.nodemanager.resource.memory.enabled)
> is disabled, leaving container memory unbounded
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-11967
> URL: https://issues.apache.org/jira/browse/YARN-11967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 3.2.4, 3.3.6, 3.5.0, 3.4.3
> Reporter: yanbin.zhang
> Assignee: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
>
> h3. Problem
> When the CGroups memory controller is *not* enabled
> (\{{yarn.nodemanager.resource.memory.enabled=false}}, the default) and the
> operator relies on the legacy polling-based memory check
> (\{{yarn.nodemanager.pmem-check-enabled=true}}), containers that exceed their
> requested memory are *not* killed. A container that requested e.g. 4 GB (a
> Spark executor) can grow to 20 GB+ without being terminated, eventually
> causing NodeManager / host memory pressure.
> h3. Root cause
> {\{ContainersMonitorImpl#checkLimit()}} short-circuits the polling-based
> pmem/vmem check whenever strict memory enforcement is on and elastic memory
> control is off:
> {code:java}
> if (strictMemoryEnforcement && !elasticMemoryEnforcement) {
> // the oom-kill would take care of it.
> return;
> }
> {code}
> This skip logic was introduced by YARN-8930. The intent is correct: when
> CGroups-based strict memory enforcement is active, the kernel OOM killer
> enforces the hard limit, so the polling check is redundant.
> The problem is that \{{strictMemoryEnforcement}} is derived *only* from
> {\{yarn.nodemanager.resource.memory.enforced}}, whose default is \{{true}}:
> {code:java}
> strictMemoryEnforcement = conf.getBoolean(
> YarnConfiguration.NM_MEMORY_RESOURCE_ENFORCED, // default: true
> YarnConfiguration.DEFAULT_NM_MEMORY_RESOURCE_ENFORCED);
> {code}
> However, the CGroups memory hard limit is *only written* when
> {\{yarn.nodemanager.resource.memory.enabled}} is \{{true}} (default
> \{{false}}),
> as seen in \{{ResourceHandlerModule#initMemoryResourceHandler()}}:
> {code:java}
> if (conf.getBoolean(NM_MEMORY_RESOURCE_ENABLED,
> DEFAULT_NM_MEMORY_RESOURCE_ENABLED)) {
> return getCgroupsMemoryResourceHandler(conf); // writes the hard limit
> }
> return null;
> {code}
> So with the default configuration (\{{memory.enabled=false}},
> {\{memory.enforced=true}}):
> ||Mechanism||State||Result||
> |CGroups hard limit + OOM killer|not written (memory controller disabled)|not
> enforced|
> |Polling pmem/vmem check|short-circuited because
> strictMemoryEnforcement==true|skipped|
> => *No mechanism limits container memory.* This is inconsistent: the CGroups
> memory handler itself is correctly gated by \{{memory.enabled}}, but the
> monitor's skip logic is not.
> h3. Reproduce
> # \{{yarn.nodemanager.container-executor.class}} =
> \{{DefaultContainerExecutor}}
> (or LCE without enabling CGroups memory).
> # Do *not* set \{{yarn.nodemanager.resource.memory.enabled}} (defaults to
> \{{false}}).
> # \{{yarn.nodemanager.pmem-check-enabled=true}}.
> # Leave \{{yarn.nodemanager.resource.memory.enforced}} at its default
> (\{{true}}).
> # Launch a container (e.g. a Spark executor with 4 GB) that allocates far more
> memory than requested.
> # Expected: container killed with \{{KILLED_EXCEEDED_PMEM}}. Actual: the
> container keeps growing and is never killed.
> h3. Fix
> Gate \{{strictMemoryEnforcement}} on *both* \{{memory.enabled}} and
> {\{memory.enforced}}, so the polling check is skipped only when CGroups memory
> enforcement is actually in effect.
> h3. Impact / Compatibility
> * Clusters that already enabled CGroups memory (\{{memory.enabled=true}},
> {\{memory.enforced=true}}) are *unaffected* -- behaviour is identical (skip
> polling, rely on the OOM killer).
> * Clusters that did *not* enable CGroups memory but rely on pmem/vmem polling
> now get the protection they expect.
> * No configuration change required; the default value of \{{memory.enforced}}
> remains \{{true}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
