This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ad9c5f81e222 [SPARK-60837][DOC] Document that NodeManagers must run on
a supported Java version for the Spark Shuffle Service
ad9c5f81e222 is described below
commit ad9c5f81e22244990fc2676d6007e2aa27812e80
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 01:15:32 2026 -0800
[SPARK-60837][DOC] Document that NodeManagers must run on a supported Java
version for the Spark Shuffle Service
### What changes were proposed in this pull request?
This PR adds a paragraph to the `Configuring the External Shuffle Service`
section of `docs/running-on-yarn.md`. It states that:
- the Spark Shuffle Service runs inside the `NodeManager` JVM, so all
`NodeManager`s must run on a supported version of Java, and
- unlike Spark applications, the shuffle service cannot use a different JDK
from the `NodeManager`.
The paragraph links the supported Java versions in `docs/index.md` instead
of naming a Java version, so the same text stays correct when the minimum Java
version changes.
### Why are the changes needed?
`spark-<version>-yarn-shuffle.jar` is built from `common/network-yarn`,
which does not override `java.version`. So its classes target the minimum Java
version of Apache Spark, which is Java 17 since Apache Spark 4.0.0. For
example, `YarnShuffleService.class` has the class file major version 61 in the
4.0.4, 4.1.3, 4.2.0, and 4.3.0 binary distributions. A `NodeManager` on an
older Java cannot load `org.apache.spark.network.yarn.YarnShuffleService`.
The guide explains how to use a different JDK for Spark applications when
the YARN cluster runs an older Java (`Configuring different JDKs for Spark
Applications`). That sets `JAVA_HOME` of the AM and the executors only, and
does not help the shuffle service, which the `NodeManager` loads as an
auxiliary service. However, the guide has never mentioned the Java requirement
of the `NodeManager`s. For example, the
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) [...]
This was raised in the review of #59897
(https://github.com/apache/spark/pull/59897#discussion_r4452535582), which
makes Java 21 the minimum in Apache Spark 5.0.0. The documentation gap is
independent of #59897 because it exists since Apache Spark 4.0.0.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Manual review.
- Both link targets already exist: the `Downloading` section of `index.md`,
which lists the supported Java versions, and the `Configuring different JDKs
for Spark Applications` section, which the same page already links at the top.
- The class file versions above were read from
`yarn/spark-<version>-yarn-shuffle.jar` of the binary distributions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60052 from dongjoon-hyun/SPARK-60837.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 152e072adb20172a27b595f5a36d8b53df91986b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/running-on-yarn.md | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index fa689c1e18f1..34adda73daad 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -874,6 +874,11 @@ will include a list of all tokens obtained, and their
expiry details
# Configuring the External Shuffle Service
+The Spark Shuffle Service runs inside the `NodeManager` JVM, so all
`NodeManager`s in your cluster
+must run on a [supported version of Java](index.html#downloading). Unlike
Spark applications, the
+shuffle service cannot use a different JDK from the `NodeManager` (see
+[Configuring different JDKs for Spark
Applications](#configuring-different-jdks-for-spark-applications)).
+
To start the Spark Shuffle Service on each `NodeManager` in your YARN cluster,
follow these
instructions:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]