peter-toth commented on code in PR #59897: URL: https://github.com/apache/spark/pull/59897#discussion_r4452535582
########## docs/running-on-yarn.md: ########## @@ -33,7 +33,7 @@ Please see [Spark Security](security.html) and the specific security sections in # Launching Spark on YARN -Apache Hadoop supports Java 17 since 3.5.0, while Apache Spark requires at least Java 17 since 4.0.0. When running on a YARN cluster whose Hadoop version is older than 3.5.0, a different JDK should be configured for Spark applications. +Apache Spark requires at least Java 21 since 5.0.0. When running on a YARN cluster with a Java version older than 21, a different JDK should be configured for Spark applications. Review Comment: **Finding 1.** The YARN external shuffle service is compiled for Java 21 as well, since `common/network-yarn` doesn't override `java.version`. It doesn't run in a Spark container. It runs inside the NodeManager JVM, since the "Configuring the External Shuffle Service" section adds `spark-<version>-yarn-shuffle.jar` to the NodeManager classpath. So setting `JAVA_HOME` for the AM and the executors doesn't help it, and a NodeManager on Java 17 can't load `YarnShuffleService` (class file version 65). The guide used to describe running YARN itself on Java 17 with Hadoop 3.5.0, so such clusters are likely. Could this paragraph say so? For example: ```suggestion Apache Spark requires at least Java 21 since 5.0.0. When running on a YARN cluster with a Java version older than 21, a different JDK should be configured for Spark applications. This does not apply to the [external shuffle service](#configuring-the-external-shuffle-service), which runs inside the NodeManager JVM, so the NodeManagers must run on Java 21 or later to use it. ``` It would also fit the user-facing list in the PR description, next to `spark-launcher`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
