This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 099387673f17 [SPARK-60837][DOC] Document that NodeManagers must run on 
a supported Java version for the Spark Shuffle Service
099387673f17 is described below

commit 099387673f17c37054e8f19c9e33736f8a3cc592
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 01:15:32 2026 -0800

    [SPARK-60837][DOC] Document that NodeManagers must run on a supported Java 
version for the Spark Shuffle Service
    
    ### What changes were proposed in this pull request?
    
    This PR adds a paragraph to the `Configuring the External Shuffle Service` 
section of `docs/running-on-yarn.md`. It states that:
    - the Spark Shuffle Service runs inside the `NodeManager` JVM, so all 
`NodeManager`s must run on a supported version of Java, and
    - unlike Spark applications, the shuffle service cannot use a different JDK 
from the `NodeManager`.
    
    The paragraph links the supported Java versions in `docs/index.md` instead 
of naming a Java version, so the same text stays correct when the minimum Java 
version changes.
    
    ### Why are the changes needed?
    
    `spark-<version>-yarn-shuffle.jar` is built from `common/network-yarn`, 
which does not override `java.version`. So its classes target the minimum Java 
version of Apache Spark, which is Java 17 since Apache Spark 4.0.0. For 
example, `YarnShuffleService.class` has the class file major version 61 in the 
4.0.4, 4.1.3, 4.2.0, and 4.3.0 binary distributions. A `NodeManager` on an 
older Java cannot load `org.apache.spark.network.yarn.YarnShuffleService`.
    
    The guide explains how to use a different JDK for Spark applications when 
the YARN cluster runs an older Java (`Configuring different JDKs for Spark 
Applications`). That sets `JAVA_HOME` of the AM and the executors only, and 
does not help the shuffle service, which the `NodeManager` loads as an 
auxiliary service. However, the guide has never mentioned the Java requirement 
of the `NodeManager`s. For example, the 
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19) [...]
    
    This was raised in the review of #59897 
(https://github.com/apache/spark/pull/59897#discussion_r4452535582), which 
makes Java 21 the minimum in Apache Spark 5.0.0. The documentation gap is 
independent of #59897 because it exists since Apache Spark 4.0.0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a documentation-only change.
    
    ### How was this patch tested?
    
    Manual review.
    - Both link targets already exist: the `Downloading` section of `index.md`, 
which lists the supported Java versions, and the `Configuring different JDKs 
for Spark Applications` section, which the same page already links at the top.
    - The class file versions above were read from 
`yarn/spark-<version>-yarn-shuffle.jar` of the binary distributions.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60052 from dongjoon-hyun/SPARK-60837.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 152e072adb20172a27b595f5a36d8b53df91986b)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/running-on-yarn.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 465f3a9d075a..46761899d08f 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -850,6 +850,11 @@ will include a list of all tokens obtained, and their 
expiry details
 
 # Configuring the External Shuffle Service
 
+The Spark Shuffle Service runs inside the `NodeManager` JVM, so all 
`NodeManager`s in your cluster
+must run on a [supported version of Java](index.html#downloading). Unlike 
Spark applications, the
+shuffle service cannot use a different JDK from the `NodeManager` (see
+[Configuring different JDKs for Spark 
Applications](#configuring-different-jdks-for-spark-applications)).
+
 To start the Spark Shuffle Service on each `NodeManager` in your YARN cluster, 
follow these
 instructions:
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to