dongjoon-hyun opened a new pull request, #60052: URL: https://github.com/apache/spark/pull/60052
### What changes were proposed in this pull request? This PR adds a paragraph to the `Configuring the External Shuffle Service` section of `docs/running-on-yarn.md`. It states that: - the Spark Shuffle Service runs inside the `NodeManager` JVM, so all `NodeManager`s must run on a supported version of Java, and - unlike Spark applications, the shuffle service cannot use a different JDK from the `NodeManager`. The paragraph links the supported Java versions in `docs/index.md` instead of naming a Java version, so the same text stays correct when the minimum Java version changes. ### Why are the changes needed? `spark-<version>-yarn-shuffle.jar` is built from `common/network-yarn`, which does not override `java.version`. So its classes target the minimum Java version of Apache Spark, which is Java 17 since Apache Spark 4.0.0. For example, `YarnShuffleService.class` has the class file major version 61 in the 4.0.4, 4.1.3, 4.2.0, and 4.3.0 binary distributions. A `NodeManager` on an older Java cannot load `org.apache.spark.network.yarn.YarnShuffleService`. The guide explains how to use a different JDK for Spark applications when the YARN cluster runs an older Java (`Configuring different JDKs for Spark Applications`). That sets `JAVA_HOME` of the AM and the executors only, and does not help the shuffle service, which the `NodeManager` loads as an auxiliary service. However, the guide has never mentioned the Java requirement of the `NodeManager`s. For example, the [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) guide said `Apache Hadoop does not support Java 17 as of 3.4.1`, while its `Configuring the External Shuffle Service` section had no Java requirement. This was raised in the review of #59897 (https://github.com/apache/spark/pull/59897#discussion_r4452535582), which makes Java 21 the minimum in Apache Spark 5.0.0. The documentation gap is independent of #59897 because it exists since Apache Spark 4.0.0. ### Does this PR introduce _any_ user-facing change? No. This is a documentation-only change. ### How was this patch tested? Manual review. - Both link targets already exist: the `Downloading` section of `index.md`, which lists the supported Java versions, and the `Configuring different JDKs for Spark Applications` section, which the same page already links at the top. - The class file versions above were read from `yarn/spark-<version>-yarn-shuffle.jar` of the binary distributions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
