HyukjinKwon opened a new pull request, #57017: URL: https://github.com/apache/spark/pull/57017
> **[DO-NOT-MERGE]** — this PR is a CI-fix verification opened from a personal fork by an automated multi-agent session. It is here to prove the fix greens the flaky `yarn` lane before a real PR is filed. Do not merge as-is. ### What changes were proposed in this pull request? Follow-up to [SPARK-57710](https://issues.apache.org/jira/browse/SPARK-57710) / [SPARK-57650](https://issues.apache.org/jira/browse/SPARK-57650). Those changes tried to stop the recurring `YarnClusterSuite` timeouts by giving the test mini `NodeManager` more memory: ```scala yarnConf.setInt("yarn.nodemanager.resource.memory-mb", 8192) yarnConf.setInt("yarn.scheduler.maximum-allocation-mb", 8192) ``` The `yarn.scheduler.maximum-allocation-mb` part takes effect, but **`yarn.nodemanager.resource.memory-mb` is silently ignored by `MiniYARNCluster`**. Its `NodeManagerWrapper.serviceInit` (hadoop-yarn-server-tests) does: ```java config.setInt(NM_PMEM_MB, config.getInt( YarnConfiguration.YARN_MINICLUSTER_NM_PMEM_MB, // "yarn.minicluster.yarn.nodemanager.resource.memory-mb" YarnConfiguration.DEFAULT_YARN_MINICLUSTER_NM_PMEM_MB)); // 4 * 1024 = 4096 ``` i.e. it unconditionally overwrites `yarn.nodemanager.resource.memory-mb` with the value of the **minicluster-prefixed** key, which defaults to 4096. So the mini NM only ever advertised ~4GB, not 8GB. This PR sets the key that `MiniYARNCluster` actually reads: ```scala yarnConf.setInt("yarn.minicluster.yarn.nodemanager.resource.memory-mb", 8192) ``` (The pre-existing `yarn.nodemanager.resource.memory-mb` line is kept — harmless, and keeps the RM's view consistent.) Test-only change. ### Why are the changes needed? The scheduled `Build / Java21`, `Build / Java25` and `Build / Maven (JDK 17/21)` master lanes still fail in the `yarn` module ~30-40% of runs, always the same six `YarnClusterSuite` cluster-mode tests timing out after 3 minutes: ``` The code passed to eventually never returned normally. Attempted 190 times over 3.0 minutes. Last failure message: handle.getState().isFinal() was false. (BaseYarnClusterSuite.scala:228) ``` The `yarn-app-log` artifact of a failing run (`28752150710`) is conclusive: - `__spark_conf__.properties` contains `yarn.minicluster.yarn.nodemanager.resource.memory-mb=4096`. - AM container `stderr` reports `Cluster resources: <memory:1024, vCores:6>` (also `<memory:0>` / `<memory:2048>`) — never the intended 8192. - With only ~4GB, once the ~1.4GB AM plus a prior test's containers are running, the next app's executors (2 × 1408MB) cannot be scheduled; the app never reaches a final state and the suite times out. Sizing the mini NM via the correct key gives the AM plus a few executors real headroom, removing the starvation race. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? The `yarn` module was run repeatedly on a fork via GitHub Actions (the failure was intermittent, so a single green run is not sufficient). Run links are added as comments on this PR as they complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
