yanbin.zhang created YARN-11967:
-----------------------------------
Summary: ContainersMonitor skips the polling-based memory check
when memory.enforced is true even though CGroups memory
(yarn.nodemanager.resource.memory.enabled) is disabled, leaving container
memory unbounded
Key: YARN-11967
URL: https://issues.apache.org/jira/browse/YARN-11967
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.4.3, 3.5.0, 3.3.6, 3.2.4
Reporter: yanbin.zhang
#### Problem
When the CGroups memory controller is **not** enabled
(`yarn.nodemanager.resource.memory.enabled=false`, the default) and the
operator relies on the legacy polling-based memory check
(`yarn.nodemanager.pmem-check-enabled=true`), containers that exceed their
requested memory are **not** killed. A container that requested e.g. 4 GB
(Spark executor) can grow to 20 GB+ without being terminated, eventually
causing NodeManager / host memory pressure.
#### Root cause
`ContainersMonitorImpl#checkLimit()` short-circuits the polling-based
pmem/vmem check whenever strict memory enforcement is on and elastic memory
control is off:
```java
if (strictMemoryEnforcement && !elasticMemoryEnforcement) {
// the oom-kill would take care of it.
return;
}
```
This skip logic was introduced by **YARN-8930**. The intent is correct: when
CGroups-based strict memory enforcement is active, the kernel OOM killer
enforces the hard limit, so the polling check is redundant.
The problem is that `strictMemoryEnforcement` is derived **only** from
`yarn.nodemanager.resource.memory.enforced`, whose default is `true`:
```java
strictMemoryEnforcement = conf.getBoolean(
YarnConfiguration.NM_MEMORY_RESOURCE_ENFORCED, // default: true
YarnConfiguration.DEFAULT_NM_MEMORY_RESOURCE_ENFORCED);
```
However, the CGroups memory hard limit is **only written** when
`yarn.nodemanager.resource.memory.enabled` is `true` (default `false`), as
seen in `ResourceHandlerModule#initMemoryResourceHandler()`:
```java
if (conf.getBoolean(NM_MEMORY_RESOURCE_ENABLED,
DEFAULT_NM_MEMORY_RESOURCE_ENABLED)) {
return getCgroupsMemoryResourceHandler(conf); // writes memory.limit_in_bytes /
memory.max
}
return null;
```
So with the default configuration (`memory.enabled=false`,
`memory.enforced=true`):
| Mechanism | State | Result |
|---|---|---|
| CGroups hard limit + OOM killer | not written (memory controller disabled) |
not enforced |
| Polling pmem/vmem check | short-circuited because
`strictMemoryEnforcement==true` | skipped |
=> **No mechanism limits container memory.** This is inconsistent: the CGroups
memory handler itself is correctly gated by `memory.enabled`, but the monitor's
skip logic is not.
#### Reproduce
1. `yarn.nodemanager.container-executor.class` = `DefaultContainerExecutor`
(or LCE without enabling CGroups memory).
2. Do **not** set `yarn.nodemanager.resource.memory.enabled` (defaults to
`false`).
3. `yarn.nodemanager.pmem-check-enabled=true`.
4. Leave `yarn.nodemanager.resource.memory.enforced` at its default (`true`).
5. Launch a container (e.g. Spark executor with 4 GB) that allocates far more
memory than requested.
6. Expected: container killed with `KILLED_EXCEEDED_PMEM`.
Actual: container keeps growing, never killed.
#### Fix
Gate `strictMemoryEnforcement` on **both** `memory.enabled` and
`memory.enforced`, so the polling check is only skipped when CGroups memory
enforcement is actually in effect.
#### Impact / Compatibility
- Clusters that already enabled CGroups memory
(`memory.enabled=true`, `memory.enforced=true`) are **unaffected** — behaviour
is identical (skip polling, rely on OOM killer).
- Clusters that did **not** enable CGroups memory but rely on pmem/vmem polling
now get the protection they expect.
- No configuration change required; default value of `memory.enforced`
remains `true`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
