This is an automated email from the ASF dual-hosted git repository. HyukjinKwon pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/spark.git
commit d286518d8af12ffc0f6ca9e8c6e33f9affd6d406 Author: Hyukjin Kwon <[email protected]> AuthorDate: Mon Jul 6 12:14:49 2026 +0900 [SPARK-57710][YARN][TEST][FOLLOWUP] Size YarnClusterSuite mini NodeManager via the yarn.minicluster.* key ### What changes were proposed in this pull request? Follow-up to [SPARK-57710](https://issues.apache.org/jira/browse/SPARK-57710) / [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650). Those changes tried to stop the recurring `YarnClusterSuite` timeouts by giving the test mini `NodeManager` more memory: ```scala yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192) yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192) ``` The `yarn.scheduler.maximum-allocation-mb` part takes effect, but **`yarn.nodemanager.resource.memory-mb` is silently ignored by `MiniYARNCluster`**. Its `NodeManagerWrapper.serviceInit` (hadoop-yarn-server-tests) does: ```java config.setInt(NM_PMEM_MB, config.getInt( YarnConfiguration.YARN_MINICLUSTER_NM_PMEM_MB, // "yarn.minicluster.yarn.nodemanager.resource.memory-mb" YarnConfiguration.DEFAULT_YARN_MINICLUSTER_NM_PMEM_MB)); // 4 * 1024 = 4096 ``` i.e. it unconditionally overwrites `yarn.nodemanager.resource.memory-mb` with the value of the **minicluster-prefixed** key, which defaults to 4096. So the mini NM only ever advertised ~4GB, not 8GB. This PR sets the key that `MiniYARNCluster` actually reads: ```scala yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb", 8192) ``` (The pre-existing `yarn.nodemanager.resource.memory-mb` line is kept — harmless, and keeps the RM's view consistent.) Test-only change. ### Why are the changes needed? The scheduled `Build / Java21`, `Build / Java25` and `Build / Maven (JDK 17/21)` master lanes still fail in the `yarn` module ~30-40% of runs, always the same six `YarnClusterSuite` cluster-mode tests timing out after 3 minutes: ``` The code passed to eventually never returned normally. Attempted 190 times over 3.0 minutes. Last failure message: handle.getState().isFinal() was false. (BaseYarnClusterSuite.scala:228) ``` The `yarn-app-log` artifact of a failing run (`28752150710`) is conclusive: - `__spark_conf__.properties` contains `yarn.minicluster.yarn.nodemanager.resource.memory-mb=4096`. - AM container `stderr` reports `Cluster resources: <memory:1024, vCores:6>` (also `<memory:0>` / `<memory:2048>`) — never the intended 8192. - With only ~4GB, once the ~1.4GB AM plus a prior test's containers are running, the next app's executors (2 × 1408MB) cannot be scheduled; the app never reaches a final state and the suite times out. Sizing the mini NM via the correct key gives the AM plus a few executors real headroom, removing the starvation race. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? The `yarn` module was run repeatedly on a fork via GitHub Actions. Because the failure is intermittent (the six cluster-mode tests timed out in ~30-40% of unpatched runs), multiple green runs are collected rather than one. **Failed (before this change — apache/spark `master`, unpatched):** - `Build / Java25` -> `yarn` module FAILED: https://github.com/apache/spark/actions/runs/28752150710/job/85254025298 - `Build / Maven (JDK 17)` -> `yarn` module FAILED: https://github.com/apache/spark/actions/runs/28742560271/job/85231134021 **Passed (with this change — fork verification):** - `yarn` module PASSED, sample 1: https://github.com/apache/spark/actions/runs/28757193236 — `YarnClusterSuite` `tests=30, failures=0, errors=0, skipped=0`; full module `tests=243, failures=0`. Ran in ~17 min vs. the ~40 min drag of the timing-out failures. - `yarn` module PASSED, sample 2: https://github.com/apache/spark/actions/runs/28758958288 — `YarnClusterSuite` again `tests=30, failures=0, errors=0`. - Repeat validation (independent verification that converged on the same fix): 6 consecutive full `YarnClusterSuite` runs on JDK 17, all green ([run 28757813841](https://github.com/HyukjinKwon/spark/actions/runs/28757813841)) — 6/6 ✅. Given the original failure reproduced ~30-40% of the time, these consecutive green runs are strong evidence the mini-NodeManager memory starvation race is gone. The six formerly-timing-out cluster-mode tests pass in every run. Closes #57017 from HyukjinKwon/ci-fix/tmp5-yarn-minicluster-mem. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]> (cherry picked from commit f4fc59d906782870c1e8f67cdb5472f9725b2e87) --- .../scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala index 5112e62d838c..e9484af9663a 100644 --- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala +++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala @@ -116,6 +116,13 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers { // headroom left for the executors these tests request. On a busy CI runner that makes // executor allocation slow/racy and the YarnClusterSuite apps time out waiting to finish. // The CI hosts have plenty of RAM, so let the NM offer enough for the AM plus a few executors. + // + // NOTE: MiniYARNCluster ignores a plain `yarn.nodemanager.resource.memory-mb`. Its + // NodeManager.serviceInit unconditionally overwrites that key with + // yarn.minicluster.yarn.nodemanager.resource.memory-mb (default 4096) + // so the minicluster-prefixed key is the one that actually sizes the mini NM. Set both: the + // prefixed key is what takes effect, and the plain key keeps the RM's view consistent. + yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb", 8192) yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192) yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
