This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f4fc59d90678 [SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite
mini NodeManager via the yarn.minicluster.* key
f4fc59d90678 is described below
commit f4fc59d906782870c1e8f67cdb5472f9725b2e87
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Jul 6 12:14:49 2026 +0900
[SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager
via the yarn.minicluster.* key
### What changes were proposed in this pull request?
Follow-up to
[SPARK-57710](https://issues.apache.org/jira/browse/SPARK-57710) /
[SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650). Those changes
tried to stop the recurring `YarnClusterSuite` timeouts by giving the test mini
`NodeManager` more memory:
```scala
yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
```
The `yarn.scheduler.maximum-allocation-mb` part takes effect, but
**`yarn.nodemanager.resource.memory-mb` is silently ignored by
`MiniYARNCluster`**. Its `NodeManagerWrapper.serviceInit`
(hadoop-yarn-server-tests) does:
```java
config.setInt(NM_PMEM_MB, config.getInt(
YarnConfiguration.YARN_MINICLUSTER_NM_PMEM_MB, //
"yarn.minicluster.yarn.nodemanager.resource.memory-mb"
YarnConfiguration.DEFAULT_YARN_MINICLUSTER_NM_PMEM_MB)); // 4 * 1024 =
4096
```
i.e. it unconditionally overwrites `yarn.nodemanager.resource.memory-mb`
with the value of the **minicluster-prefixed** key, which defaults to 4096. So
the mini NM only ever advertised ~4GB, not 8GB.
This PR sets the key that `MiniYARNCluster` actually reads:
```scala
yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb",
8192)
```
(The pre-existing `yarn.nodemanager.resource.memory-mb` line is kept —
harmless, and keeps the RM's view consistent.) Test-only change.
### Why are the changes needed?
The scheduled `Build / Java21`, `Build / Java25` and `Build / Maven (JDK
17/21)` master lanes still fail in the `yarn` module ~30-40% of runs, always
the same six `YarnClusterSuite` cluster-mode tests timing out after 3 minutes:
```
The code passed to eventually never returned normally. Attempted 190 times
over 3.0 minutes.
Last failure message: handle.getState().isFinal() was false.
(BaseYarnClusterSuite.scala:228)
```
The `yarn-app-log` artifact of a failing run (`28752150710`) is conclusive:
- `__spark_conf__.properties` contains
`yarn.minicluster.yarn.nodemanager.resource.memory-mb=4096`.
- AM container `stderr` reports `Cluster resources: <memory:1024,
vCores:6>` (also `<memory:0>` / `<memory:2048>`) — never the intended 8192.
- With only ~4GB, once the ~1.4GB AM plus a prior test's containers are
running, the next app's executors (2 × 1408MB) cannot be scheduled; the app
never reaches a final state and the suite times out.
Sizing the mini NM via the correct key gives the AM plus a few executors
real headroom, removing the starvation race.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
The `yarn` module was run repeatedly on a fork via GitHub Actions. Because
the failure is intermittent (the six cluster-mode tests timed out in ~30-40% of
unpatched runs), multiple green runs are collected rather than one.
**Failed (before this change — apache/spark `master`, unpatched):**
- `Build / Java25` -> `yarn` module FAILED:
https://github.com/apache/spark/actions/runs/28752150710/job/85254025298
- `Build / Maven (JDK 17)` -> `yarn` module FAILED:
https://github.com/apache/spark/actions/runs/28742560271/job/85231134021
**Passed (with this change — fork verification):**
- `yarn` module PASSED, sample 1:
https://github.com/apache/spark/actions/runs/28757193236 — `YarnClusterSuite`
`tests=30, failures=0, errors=0, skipped=0`; full module `tests=243,
failures=0`. Ran in ~17 min vs. the ~40 min drag of the timing-out failures.
- `yarn` module PASSED, sample 2:
https://github.com/apache/spark/actions/runs/28758958288 — `YarnClusterSuite`
again `tests=30, failures=0, errors=0`.
- Repeat validation (independent verification that converged on the same
fix): 6 consecutive full `YarnClusterSuite` runs on JDK 17, all green ([run
28757813841](https://github.com/HyukjinKwon/spark/actions/runs/28757813841)) —
6/6 ✅.
Given the original failure reproduced ~30-40% of the time, these
consecutive green runs are strong evidence the mini-NodeManager memory
starvation race is gone. The six formerly-timing-out cluster-mode tests pass in
every run.
Closes #57017 from HyukjinKwon/ci-fix/tmp5-yarn-minicluster-mem.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 7 +++++++
1 file changed, 7 insertions(+)
diff --git
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
index 5112e62d838c..e9484af9663a 100644
---
a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
+++
b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala
@@ -116,6 +116,13 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite
with Matchers {
// headroom left for the executors these tests request. On a busy CI
runner that makes
// executor allocation slow/racy and the YarnClusterSuite apps time out
waiting to finish.
// The CI hosts have plenty of RAM, so let the NM offer enough for the AM
plus a few executors.
+ //
+ // NOTE: MiniYARNCluster ignores a plain
`yarn.nodemanager.resource.memory-mb`. Its
+ // NodeManager.serviceInit unconditionally overwrites that key with
+ // yarn.minicluster.yarn.nodemanager.resource.memory-mb (default 4096)
+ // so the minicluster-prefixed key is the one that actually sizes the mini
NM. Set both: the
+ // prefixed key is what takes effect, and the plain key keeps the RM's
view consistent.
+ yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb",
8192)
yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192)
yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
