This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 21c599680a83 [SPARK-60669][BUILD][INFRA][DOC] Drop Java 17 Support
21c599680a83 is described below
commit 21c599680a83bfc76e8c1da6efff1e2f10e0c123
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 08:25:37 2026 -0800
[SPARK-60669][BUILD][INFRA][DOC] Drop Java 17 Support
### What changes were proposed in this pull request?
This PR aims to drop Java 17 support in Apache Spark 5.0.0 by making Java
21 the minimum Java version.
- `pom.xml`: Change `java.version`, `java.minimum.version`, and the
`-release` option of `scala-maven-plugin` to 21.
- GitHub Actions: Use Java 21 instead of Java 17 for `master`.
`publish_snapshot.yml` keeps Java 17 for `branch-4.*`.
- GitHub Actions: Turn `build_java17.yml` and `build_maven_java21.yml` into
no-op placeholders on `master` like `build_scala213.yml`, and remove the
`master` badge of `build_maven_java21.yml` from `README.md`. `master` no longer
supports Java 17, and `build_maven.yml` now runs the same Maven build with Java
21. The files are kept because the branch schedulers dispatch them for
`branch-4.*`.
- Release: Install Java 21 in the `master` release image, and make
`release-build.sh` reject Java versions older than 21 for Apache Spark 5.0.0
and later.
- SparkR: Require Java 21 in `SystemRequirements`, which
`checkJavaVersion()` reads.
- `bin/pyspark`: Install JDK 21 instead of JDK 17 when no Java is found.
- K8s integration tests: Use Java 21 by default in
`dev-run-integration-tests.sh`.
- Docs: Update the Java version in `README.md`, `docs/index.md`,
`docs/building-spark.md`, `docs/running-on-yarn.md`, `docs/security.md`,
`docs/tuning.md`, and `python/docs/source/getting_started/install.rst`.
### Why are the changes needed?
The minimum supported Java version for Apache Spark 5.0.0 is Java 21.
### Does this PR introduce _any_ user-facing change?
Yes. Apache Spark 5.0.0 requires Java 21 or later. Since all Spark
artifacts are compiled for Java 21 (class file version 65), this applies to the
client side as well as the cluster runtime. For example, Java 21 is required
for:
- Applications using the Spark Connect JVM client
(`spark-connect-client-jvm`) or the Spark Connect JDBC driver
(`spark-connect-client-jdbc`).
- Services embedding `spark-launcher`.
- Libraries compiled against Apache Spark 5.0.0 artifacts.
The pure-Python Spark Connect client (`pyspark-client`) is not affected.
### How was this patch tested?
Pass the CIs. In addition, I manually verified the following.
- Maven and SBT reject Java 17, and `Test/compile`, `unidoc`, and MiMa pass
with Java 21.
- `openjdk-21-jdk-headless` installs on the base image of the release image
(`ubuntu:jammy-20250819`), and `release-build.sh` accepts it for
`5.0.0-SNAPSHOT` while rejecting Java 17 for Apache Spark 5.0.0.
- All workflow files parse, their names are unique, and every workflow that
the branch schedulers dispatch still has `workflow_dispatch`.
- The new `SystemRequirements` of SparkR, parsed the same way as
`checkJavaVersion()`, rejects Java 17 and accepts Java 21 and 25.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59897 from dongjoon-hyun/SPARK-60669.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.github/workflows/benchmark.yml | 5 ++-
.github/workflows/build_and_test.yml | 4 +--
.github/workflows/build_codegen_jdk.yml | 4 +--
.github/workflows/build_coverage.yml | 2 +-
.github/workflows/build_java17.yml | 37 ++++++++--------------
.github/workflows/build_maven.yml | 2 +-
.github/workflows/build_maven_java21.yml | 25 +++++++++------
.github/workflows/build_non_ansi.yml | 4 +--
.github/workflows/build_python_3.11.yml | 2 +-
.../workflows/build_python_3.12_classic_only.yml | 2 +-
.github/workflows/build_python_3.12_pandas_3.yml | 2 +-
.github/workflows/build_python_3.13.yml | 2 +-
.github/workflows/build_python_3.14.yml | 2 +-
.github/workflows/build_python_3.14_nogil.yml | 2 +-
.github/workflows/build_python_3.15.yml | 2 +-
.github/workflows/build_python_connect.yml | 2 +-
.github/workflows/build_python_connect40.yml | 2 +-
.github/workflows/build_python_minimum.yml | 2 +-
.github/workflows/build_rockdb_as_ui_backend.yml | 4 +--
.github/workflows/build_sparkr_window.yml | 4 +--
.github/workflows/build_uds.yml | 4 +--
.github/workflows/maven_test.yml | 2 +-
.github/workflows/pages.yml | 4 +--
.github/workflows/publish_snapshot.yml | 10 ++++--
.github/workflows/python_hosted_runner_test.yml | 2 +-
R/pkg/DESCRIPTION | 2 +-
README.md | 3 +-
bin/pyspark | 8 ++---
dev/create-release/do-release-docker.sh | 2 +-
dev/create-release/release-build.sh | 4 +++
dev/create-release/spark-rm/Dockerfile | 6 ++--
dev/create-release/spark-rm/Dockerfile.base | 2 +-
docs/building-spark.md | 2 +-
docs/index.md | 2 +-
docs/running-on-yarn.md | 10 +++---
docs/security.md | 10 +++---
docs/tuning.md | 10 +++---
pom.xml | 6 ++--
python/docs/source/getting_started/install.rst | 2 +-
.../kubernetes/integration-tests/README.md | 2 +-
.../dev/dev-run-integration-tests.sh | 2 +-
.../datasources/v2/ffi/NativeLibrary.java | 2 +-
42 files changed, 105 insertions(+), 103 deletions(-)
diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml
index 3140cb054e57..17fbbbca5941 100644
--- a/.github/workflows/benchmark.yml
+++ b/.github/workflows/benchmark.yml
@@ -28,11 +28,10 @@ on:
default: '*'
jdk:
type: choice
- description: 'JDK version: 17, 21, or 25'
+ description: 'JDK version: 21 or 25'
required: true
- default: '17'
+ default: '21'
options:
- - '17'
- '21'
- '25'
scala:
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 126288a544b3..cff932a8fc30 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -25,7 +25,7 @@ on:
java:
required: false
type: string
- default: 17
+ default: 21
branch:
description: Branch to run the build against
required: false
@@ -953,7 +953,7 @@ jobs:
export MAVEN_CLI_OPTS="--no-transfer-progress"
./build/mvn $MAVEN_CLI_OPTS -DskipTests -Pyarn -Pkubernetes -Pvolcano
-Phive -Phive-thriftserver -Phadoop-cloud -Pjvm-profiler -Pspark-ganglia-lgpl
-Pkinesis-asl -Pcredential-aws clean install
# Native data sources are called with the FFM API, which requires Java 22
or later, so the
- # other jobs, on Java 17, skip the tests that load a library. The tests
build their library
+ # other jobs, on Java 21, skip the tests that load a library. The tests
build their library
# with the C++ compiler of the runner.
- name: Test native data sources
run: |
diff --git a/.github/workflows/build_codegen_jdk.yml
b/.github/workflows/build_codegen_jdk.yml
index 3df5f94ce484..5f49d740b978 100644
--- a/.github/workflows/build_codegen_jdk.yml
+++ b/.github/workflows/build_codegen_jdk.yml
@@ -25,7 +25,7 @@
# javac is slower than Janino, so this runs on its own schedule rather than on
# every push, skips the jobs that do not compile generated code (docs, lint,
buf,
# ui), and raises the timeout of the three jobs that no longer fit their
default.
-name: "Build / Codegen JDK backend (master, Scala 2.13, JDK 17)"
+name: "Build / Codegen JDK backend (master, Scala 2.13, JDK 21)"
on:
schedule:
@@ -40,7 +40,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
# See SPARK-57411 for the measurements behind these three.
diff --git a/.github/workflows/build_coverage.yml
b/.github/workflows/build_coverage.yml
index 9db5f5e5f971..d792753e2aa3 100644
--- a/.github/workflows/build_coverage.yml
+++ b/.github/workflows/build_coverage.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_java17.yml
b/.github/workflows/build_java17.yml
index 9b365a2a5684..e0387cb994c7 100644
--- a/.github/workflows/build_java17.yml
+++ b/.github/workflows/build_java17.yml
@@ -17,34 +17,23 @@
# under the License.
#
+# This is a placeholder. Java 17 is not supported on master since Spark 5.0, so
+# master does not run this build. The file exists here only so that the README
+# build status badge, which resolves a workflow by its path on the default
+# branch, renders for the branches that still run it (e.g. branch-4.x), and so
+# that branch4*_scheduler.yml can dispatch it to those branches with
+# `gh workflow run --ref`. The actual build is defined in the same file on
+# those branches.
+
name: "Build / Java17 (Scala 2.13, JDK 17)"
on:
workflow_dispatch:
jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/build_and_test.yml
+ noop:
if: github.repository == 'apache/spark'
- with:
- java: 17
- hadoop: hadoop3
- envs: >-
- {
- "SCALA_PROFILE": "scala2.13",
- "PYSPARK_IMAGE_TO_TEST": "",
- "PYTHON_TO_TEST": ""
- }
- jobs: >-
- {
- "build": "true",
- "sparkr": "true",
- "tpcds-1g": "true",
- "docker-integration-tests": "true",
- "k8s-integration-tests": "true",
- "oidc-e2e": "true",
- "lint" : "true"
- }
+ runs-on: ubuntu-latest
+ steps:
+ - name: No-op
+ run: echo "Java 17 is not built on this branch. See branches that
still support it (e.g. branch-4.x)."
diff --git a/.github/workflows/build_maven.yml
b/.github/workflows/build_maven.yml
index 2e40b70a6500..ee38a36579c5 100644
--- a/.github/workflows/build_maven.yml
+++ b/.github/workflows/build_maven.yml
@@ -17,7 +17,7 @@
# under the License.
#
-name: "Build / Maven (Scala 2.13, JDK 17)"
+name: "Build / Maven (Scala 2.13, JDK 21)"
on:
schedule:
diff --git a/.github/workflows/build_maven_java21.yml
b/.github/workflows/build_maven_java21.yml
index ac03fc8f4167..b69b878f0b92 100644
--- a/.github/workflows/build_maven_java21.yml
+++ b/.github/workflows/build_maven_java21.yml
@@ -17,19 +17,24 @@
# under the License.
#
-name: "Build / Maven (Scala 2.13, JDK 21)"
+# This is a placeholder. Java 21 is the default on master, so build_maven.yml
+# already runs the Maven build with Java 21 and there is no separate scheduled
+# build here. The file exists only so that the README build status badge, which
+# resolves a workflow by its path on the default branch, renders for the
+# branches that still run it as a separate build (e.g. branch-4.x), and so that
+# branch4*_scheduler.yml can dispatch it to those branches with
+# `gh workflow run --ref`. The actual build is defined in the same file on
+# those branches.
+
+name: "Build / Maven Java21 (Scala 2.13, JDK 21)"
on:
- schedule:
- - cron: '0 14 */2 * *'
workflow_dispatch:
jobs:
- run-build:
- permissions:
- packages: write
- name: Run
- uses: ./.github/workflows/maven_test.yml
+ noop:
if: github.repository == 'apache/spark'
- with:
- java: 21
+ runs-on: ubuntu-latest
+ steps:
+ - name: No-op
+ run: echo "Java 21 is the default on master and is not built
separately. See branches that run it as a separate build (e.g. branch-4.x)."
diff --git a/.github/workflows/build_non_ansi.yml
b/.github/workflows/build_non_ansi.yml
index c19081c204fd..14daac8eb2e6 100644
--- a/.github/workflows/build_non_ansi.yml
+++ b/.github/workflows/build_non_ansi.yml
@@ -17,7 +17,7 @@
# under the License.
#
-name: "Build / Non-ANSI (JDK 17, Scala 2.13)"
+name: "Build / Non-ANSI (JDK 21, Scala 2.13)"
on:
schedule:
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.11.yml
b/.github/workflows/build_python_3.11.yml
index 793f1161e948..9627e5143728 100644
--- a/.github/workflows/build_python_3.11.yml
+++ b/.github/workflows/build_python_3.11.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.12_classic_only.yml
b/.github/workflows/build_python_3.12_classic_only.yml
index 2136e877e495..d73c628c24c0 100644
--- a/.github/workflows/build_python_3.12_classic_only.yml
+++ b/.github/workflows/build_python_3.12_classic_only.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.12_pandas_3.yml
b/.github/workflows/build_python_3.12_pandas_3.yml
index 570fddfd11a5..f8d1cf915ced 100644
--- a/.github/workflows/build_python_3.12_pandas_3.yml
+++ b/.github/workflows/build_python_3.12_pandas_3.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.13.yml
b/.github/workflows/build_python_3.13.yml
index fea1d786bc6b..281225b15502 100644
--- a/.github/workflows/build_python_3.13.yml
+++ b/.github/workflows/build_python_3.13.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.14.yml
b/.github/workflows/build_python_3.14.yml
index 7b07b9b02b04..559556b06008 100644
--- a/.github/workflows/build_python_3.14.yml
+++ b/.github/workflows/build_python_3.14.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.14_nogil.yml
b/.github/workflows/build_python_3.14_nogil.yml
index 7e431b3d340e..aae911398d37 100644
--- a/.github/workflows/build_python_3.14_nogil.yml
+++ b/.github/workflows/build_python_3.14_nogil.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_3.15.yml
b/.github/workflows/build_python_3.15.yml
index 2c8535553f28..55cba6647c34 100644
--- a/.github/workflows/build_python_3.15.yml
+++ b/.github/workflows/build_python_3.15.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_python_connect.yml
b/.github/workflows/build_python_connect.yml
index b3397c3ffd16..867b2d017a3d 100644
--- a/.github/workflows/build_python_connect.yml
+++ b/.github/workflows/build_python_connect.yml
@@ -37,7 +37,7 @@ jobs:
- name: Setup build environment
uses: ./.github/actions/setup-build-env
with:
- java-version: '17'
+ java-version: '21'
- name: Install Python 3.11
uses: actions/setup-python@v7
with:
diff --git a/.github/workflows/build_python_connect40.yml
b/.github/workflows/build_python_connect40.yml
index 5e259a3a62b6..426191fb774c 100644
--- a/.github/workflows/build_python_connect40.yml
+++ b/.github/workflows/build_python_connect40.yml
@@ -39,7 +39,7 @@ jobs:
- name: Setup build environment
uses: ./.github/actions/setup-build-env
with:
- java-version: '17'
+ java-version: '21'
- name: Install Python 3.11
uses: actions/setup-python@v7
with:
diff --git a/.github/workflows/build_python_minimum.yml
b/.github/workflows/build_python_minimum.yml
index cb23a3db1454..9ad0f9d342e9 100644
--- a/.github/workflows/build_python_minimum.yml
+++ b/.github/workflows/build_python_minimum.yml
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_rockdb_as_ui_backend.yml
b/.github/workflows/build_rockdb_as_ui_backend.yml
index 506fd28e9cc1..0f1844228537 100644
--- a/.github/workflows/build_rockdb_as_ui_backend.yml
+++ b/.github/workflows/build_rockdb_as_ui_backend.yml
@@ -17,7 +17,7 @@
# under the License.
#
-name: "Build / RocksDB as UI Backend (JDK 17, Scala 2.13)"
+name: "Build / RocksDB as UI Backend (JDK 21, Scala 2.13)"
on:
schedule:
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/build_sparkr_window.yml
b/.github/workflows/build_sparkr_window.yml
index 39b0da2c0829..aa8ff04bde25 100644
--- a/.github/workflows/build_sparkr_window.yml
+++ b/.github/workflows/build_sparkr_window.yml
@@ -46,11 +46,11 @@ jobs:
key: build-sparkr-windows-maven-${{ runner.os }}-${{
hashFiles('**/pom.xml') }}
restore-keys: |
build-sparkr-windows-maven-${{ runner.os }}-
- - name: Install Java 17
+ - name: Install Java 21
uses: actions/setup-java@v6
with:
distribution: zulu
- java-version: 17
+ java-version: 21
- name: Install R 4.4.3
uses: r-lib/actions/setup-r@d3c5be51b12e724e68f33216ca3c148b66d5f0b6 #
v2.12.1
with:
diff --git a/.github/workflows/build_uds.yml b/.github/workflows/build_uds.yml
index 64a97e4faa16..e23d3acea928 100644
--- a/.github/workflows/build_uds.yml
+++ b/.github/workflows/build_uds.yml
@@ -17,7 +17,7 @@
# under the License.
#
-name: "Build / Unix Domain Socket (JDK 17, Scala 2.13)"
+name: "Build / Unix Domain Socket (JDK 21, Scala 2.13)"
on:
schedule:
@@ -32,7 +32,7 @@ jobs:
uses: ./.github/workflows/build_and_test.yml
if: github.repository == 'apache/spark'
with:
- java: 17
+ java: 21
branch: master
hadoop: hadoop3
envs: >-
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index ab720355a1e8..b83be98f947f 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -25,7 +25,7 @@ on:
java:
required: false
type: string
- default: 17
+ default: 21
branch:
description: Branch to run the build against
required: false
diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml
index 9d0d2c06f681..3b7e31f4bbfa 100644
--- a/.github/workflows/pages.yml
+++ b/.github/workflows/pages.yml
@@ -56,11 +56,11 @@ jobs:
git config --global --add safe.directory ${GITHUB_WORKSPACE}
- name: Free up disk space
run: ./dev/free_disk_space_container
- - name: Install Java 17
+ - name: Install Java 21
uses: actions/setup-java@v6
with:
distribution: zulu
- java-version: 17
+ java-version: 21
- name: Install dependencies for documentation generation
run: |
# Keep the version of Bundler here in sync with the following
locations:
diff --git a/.github/workflows/publish_snapshot.yml
b/.github/workflows/publish_snapshot.yml
index 2f10d6594f01..07b5b662b879 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish_snapshot.yml
@@ -58,12 +58,18 @@ jobs:
with:
distribution: temurin
java-version: 8
- - name: Install Java 17
- if: matrix.branch != 'branch-3.5'
+ - name: Install Java 17 for branch-4.x
+ if: matrix.branch != 'branch-3.5' && matrix.branch != 'master'
uses: actions/setup-java@v6
with:
distribution: temurin
java-version: 17
+ - name: Install Java 21
+ if: matrix.branch == 'master'
+ uses: actions/setup-java@v6
+ with:
+ distribution: temurin
+ java-version: 21
- name: Publish snapshot
env:
ASF_USERNAME: ${{ secrets.NEXUS_USER }}
diff --git a/.github/workflows/python_hosted_runner_test.yml
b/.github/workflows/python_hosted_runner_test.yml
index 696726484a39..4159cb651c66 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -25,7 +25,7 @@ on:
java:
required: false
type: string
- default: 17
+ default: 21
python:
required: false
type: string
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index a5c6876a4c93..4d16a785e0a7 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -10,7 +10,7 @@ Authors@R:
License: Apache License (== 2.0)
URL: https://www.apache.org https://spark.apache.org
BugReports: https://spark.apache.org/contributing.html
-SystemRequirements: Java (>= 17, < 26)
+SystemRequirements: Java (>= 21, < 26)
Depends:
R (>= 4.0),
methods
diff --git a/README.md b/README.md
index 133112688abe..1b0207960bb7 100644
--- a/README.md
+++ b/README.md
@@ -12,7 +12,7 @@ and Structured Streaming for stream processing.
[](https://opensource.org/licenses/Apache-2.0)
[](https://search.maven.org/search?q=g:org.apache.spark)
-[](https://adoptium.net/temurin/releases/?version=17)
+[](https://adoptium.net/temurin/releases/?version=21)
[](https://github.com/apache/spark/actions/workflows/build_main.yml)
[](https://codecov.io/gh/apache/spark)
[](https://pypi.org/project/pyspark/)
@@ -38,7 +38,6 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_uds.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_rockdb_as_ui_backend.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml)
|
-| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java25.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21_macos26.yml)
|
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21_arm.yml)
|
diff --git a/bin/pyspark b/bin/pyspark
index 42edeac4ccb0..f56b1f3cb5a7 100755
--- a/bin/pyspark
+++ b/bin/pyspark
@@ -49,7 +49,7 @@ export PYSPARK_DRIVER_PYTHON
export PYSPARK_DRIVER_PYTHON_OPTS
# Attempt to find JAVA_HOME.
-# If JAVA_HOME not set, install JDK 17 and set JAVA_HOME using a temp dir, and
adding the
+# If JAVA_HOME not set, install JDK 21 and set JAVA_HOME using a temp dir, and
adding the
# temp dir to the PYTHONPATH.
if [ -n "${JAVA_HOME}" ]; then
RUNNER="${JAVA_HOME}/bin/java"
@@ -57,19 +57,19 @@ else
if [ "$(command -v java)" ]; then
RUNNER="java"
else
- echo -n "JAVA_HOME is not set. Would you like to install JDK 17 and set
JAVA_HOME? (Y/N) " >&2
+ echo -n "JAVA_HOME is not set. Would you like to install JDK 21 and set
JAVA_HOME? (Y/N) " >&2
read -r input
if [[ "${input,,}" == "y" ]]; then
TEMP_DIR=$(mktemp -d)
$PYSPARK_DRIVER_PYTHON -m pip install --target="$TEMP_DIR" install-jdk
- export JAVA_HOME=$(PYTHONPATH="$TEMP_DIR" $PYSPARK_DRIVER_PYTHON -c
'import jdk; print(jdk.install("17"))')
+ export JAVA_HOME=$(PYTHONPATH="$TEMP_DIR" $PYSPARK_DRIVER_PYTHON -c
'import jdk; print(jdk.install("21"))')
RUNNER="${JAVA_HOME}/bin/java"
echo "JDK was installed to the path \"$JAVA_HOME\""
echo "You can avoid needing to re-install JDK by setting your
JAVA_HOME environment variable to \"$JAVA_HOME\""
else
- echo "JDK installation skipped. You can manually install JDK (17 or
later) and set JAVA_HOME in your environment."
+ echo "JDK installation skipped. You can manually install JDK (21 or
later) and set JAVA_HOME in your environment."
exit 1
fi
fi
diff --git a/dev/create-release/do-release-docker.sh
b/dev/create-release/do-release-docker.sh
index d0b4ada85659..c2ac77864301 100755
--- a/dev/create-release/do-release-docker.sh
+++ b/dev/create-release/do-release-docker.sh
@@ -45,7 +45,7 @@ Options are:
-n : dry run mode. Performs checks and local builds, but do not
upload anything.
-t [tag] : tag for the spark-rm docker image to use for building
(default: "latest").
-j [path] : path to local JDK installation to use for building. By default
the script will
- use openjdk17 installed in the docker image.
+ use openjdk21 installed in the docker image.
-s [step] : runs a single step of the process; valid steps are: tag,
build, docs, publish, finalize
EOF
}
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index 36ec915a134e..006754183343 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -689,6 +689,10 @@ if [[ $JAVA_VERSION < "1.8." ]] && [[ $SPARK_VERSION <
"4.0" ]]; then
echo "Java version $JAVA_VERSION is less than required 1.8 for 2.2+"
echo "Please set JAVA_HOME correctly."
exit 1
+elif [[ $JAVA_VERSION < "21" ]] && [[ $SPARK_VERSION > "4.99" ]]; then
+ echo "Java version $JAVA_VERSION is less than required 21 for 5.0+"
+ echo "Please set JAVA_HOME correctly."
+ exit 1
elif [[ $JAVA_VERSION < "17.0." ]] && [[ $SPARK_VERSION > "3.5.99" ]]; then
echo "Java version $JAVA_VERSION is less than required 17 for 4.0+"
echo "Please set JAVA_HOME correctly."
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index 20d8f847e1b2..e0a3a0c63a2a 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -17,14 +17,14 @@
# Spark master (5.0.0-SNAPSHOT) release image
# Extends the base image with:
-# - Java 17
+# - Java 21
# - Python 3.11 with required packages
FROM spark-rm-base:latest
-# Install Java 17 for Spark 5.x
+# Install Java 21 for Spark 5.x
RUN apt-get update && apt-get install -y \
- openjdk-17-jdk-headless \
+ openjdk-21-jdk-headless \
&& rm -rf /var/lib/apt/lists/*
# Install Python 3.11
diff --git a/dev/create-release/spark-rm/Dockerfile.base
b/dev/create-release/spark-rm/Dockerfile.base
index b7a2da7729bd..97de0e314884 100644
--- a/dev/create-release/spark-rm/Dockerfile.base
+++ b/dev/create-release/spark-rm/Dockerfile.base
@@ -24,7 +24,7 @@
# - Node.js for documentation
#
# Branch-specific Dockerfiles should use "FROM spark-rm-base:latest" and add:
-# - Java version (8 or 17)
+# - Java version (8, 17, or 21)
# - Python version and pip packages
FROM ubuntu:jammy-20250819
diff --git a/docs/building-spark.md b/docs/building-spark.md
index 4fc0724e83ff..ebe58847d517 100644
--- a/docs/building-spark.md
+++ b/docs/building-spark.md
@@ -27,7 +27,7 @@ license: |
## Apache Maven
The Maven-based build is the build of reference for Apache Spark.
-Building Spark using Maven requires Maven 3.9.16 and Java 17/21/25.
+Building Spark using Maven requires Maven 3.9.16 and Java 21/25.
Spark requires Scala 2.13; support for Scala 2.12 was removed in Spark 4.0.0.
### Setting up Maven's Memory Usage
diff --git a/docs/index.md b/docs/index.md
index a78f981ef85d..6c8acb7655e5 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -34,7 +34,7 @@ source, visit [Building Spark](building-spark.html).
Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it
should run on any platform that runs a supported version of Java. This should
include JVMs on x86_64 and ARM64. It's easy to run locally on one machine ---
all you need is to have `java` installed on your system `PATH`, or the
`JAVA_HOME` environment variable pointing to a Java installation.
-Spark runs on Java 17/21/25, Scala 2.13, Python 3.11+, and R 4.0+ (Deprecated).
+Spark runs on Java 21/25, Scala 2.13, Python 3.11+, and R 4.0+ (Deprecated).
Java 25 prior to version 25.0.3 support is deprecated as of Spark 4.2.0.
When using the Scala API, it is necessary for applications to use the same
version of Scala that Spark was compiled for. Since Spark 4.0.0, it's Scala
2.13.
diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md
index 34adda73daad..46541d0bae7d 100644
--- a/docs/running-on-yarn.md
+++ b/docs/running-on-yarn.md
@@ -33,7 +33,7 @@ Please see [Spark Security](security.html) and the specific
security sections in
# Launching Spark on YARN
-Apache Hadoop supports Java 17 since 3.5.0, while Apache Spark requires at
least Java 17 since 4.0.0. When running on a YARN cluster whose Hadoop version
is older than 3.5.0, a different JDK should be configured for Spark
applications.
+Apache Spark requires at least Java 21 since 5.0.0. When running on a YARN
cluster with a Java version older than 21, a different JDK should be configured
for Spark applications.
Please refer to [Configuring different JDKs for Spark
Applications](#configuring-different-jdks-for-spark-applications) for details.
Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory which
contains the (client side) configuration files for the Hadoop cluster.
@@ -1074,13 +1074,13 @@ process.
Note that, Spark assumes that all JVM processes runs in one application use
the same version of JDK, otherwise,
you may encounter JDK serialization issues.
-To configure a Spark application to use a JDK which has been pre-installed on
all nodes at `/opt/openjdk-17`:
+To configure a Spark application to use a JDK which has been pre-installed on
all nodes at `/opt/openjdk-21`:
- $ export JAVA_HOME=/opt/openjdk-17
+ $ export JAVA_HOME=/opt/openjdk-21
$ ./bin/spark-submit --class path.to.your.Class \
--master yarn \
- --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-17 \
- --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-17 \
+ --conf spark.yarn.appMasterEnv.JAVA_HOME=/opt/openjdk-21 \
+ --conf spark.executorEnv.JAVA_HOME=/opt/openjdk-21 \
<app jar> [app options]
Optionally, the user may want to avoid installing a different JDK on the YARN
cluster nodes, in such a case,
diff --git a/docs/security.md b/docs/security.md
index b921598b6d6b..b2c8f33ba19a 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -570,8 +570,8 @@ replaced with one of the above namespaces.
A comma-separated list of ciphers. The specified ciphers must be
supported by JVM.
<br />The reference list of protocols can be found in the "JSSE Cipher
Suite Names" section
- of the Java security guide. The list for Java 17 can be found at
- <a
href="https://docs.oracle.com/en/java/javase/17/docs/specs/security/standard-names.html#jsse-cipher-suite-names">this</a>
+ of the Java security guide. The list for Java 21 can be found at
+ <a
href="https://docs.oracle.com/en/java/javase/21/docs/specs/security/standard-names.html#jsse-cipher-suite-names">this</a>
page.
<br />Note: If not set, the default cipher suite for the JRE will be
used.
@@ -614,8 +614,8 @@ replaced with one of the above namespaces.
TLS protocol to use. The protocol must be supported by JVM.
<br />The reference list of protocols can be found in the "Additional
JSSE Standard Names"
- section of the Java security guide. For Java 17, the list can be found at
- <a
href="https://docs.oracle.com/en/java/javase/17/docs/specs/security/standard-names.html#additional-jsse-standard-names">this</a>
+ section of the Java security guide. For Java 21, the list can be found at
+ <a
href="https://docs.oracle.com/en/java/javase/21/docs/specs/security/standard-names.html#additional-jsse-standard-names">this</a>
page.
</td>
<td>ui,standalone,historyServer,rpc</td>
@@ -732,7 +732,7 @@ Or via SparkConf
"spark.hadoop.hadoop.security.credential.provider.path=jceks://
## Preparing the key stores
Key stores can be generated by `keytool` program. The reference documentation
for this tool for
-Java 17 is
[here](https://docs.oracle.com/en/java/javase/17/docs/specs/man/keytool.html).
+Java 21 is
[here](https://docs.oracle.com/en/java/javase/21/docs/specs/man/keytool.html).
The most basic steps to configure the key stores and the trust store for a
Spark Standalone
deployment mode is as follows:
diff --git a/docs/tuning.md b/docs/tuning.md
index f72dc0efd98e..e07463f204b8 100644
--- a/docs/tuning.md
+++ b/docs/tuning.md
@@ -41,12 +41,12 @@ Often, this will be the first thing you should tune to
optimize a Spark applicat
Spark aims to strike a balance between convenience (allowing you to work with
any Java type
in your operations) and performance. It provides two serialization libraries:
-* [Java
serialization](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/io/Serializable.html):
+* [Java
serialization](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/Serializable.html):
By default, Spark serializes objects using Java's `ObjectOutputStream`
framework, and can work
with any class you create that implements
-
[`java.io.Serializable`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/io/Serializable.html).
+
[`java.io.Serializable`](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/Serializable.html).
You can also control the performance of your serialization more closely by
extending
-
[`java.io.Externalizable`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/io/Externalizable.html).
+
[`java.io.Externalizable`](https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/io/Externalizable.html).
Java serialization is flexible but often quite slow, and leads to large
serialized formats for many classes.
* [Kryo serialization](https://github.com/EsotericSoftware/kryo): Spark can
also use
@@ -232,7 +232,7 @@ temporary objects created during task execution. Some steps
which may be useful
value of the JVM's `NewRatio` parameter. Many JVMs default this to 2,
meaning that the Old generation
occupies 2/3 of the heap. It should be large enough such that this fraction
exceeds `spark.memory.fraction`.
-* Since 4.0.0, Spark uses JDK 17 by default, which also makes the G1GC garbage
collector the default. Note that with
+* Spark requires JDK 21 or later, where G1GC is the default garbage collector.
Note that with
large executor heap sizes, it may be important to increase the G1 region
size with `-XX:G1HeapRegionSize`.
* As an example, if your task is reading data from HDFS, the amount of memory
used by the task can be estimated using
@@ -243,7 +243,7 @@ temporary objects created during task execution. Some steps
which may be useful
* Monitor how the frequency and time taken by garbage collection changes with
the new settings.
Our experience suggests that the effect of GC tuning depends on your
application and the amount of memory available.
-There are [many more tuning
options](https://docs.oracle.com/en/java/javase/17/gctuning/introduction-garbage-collection-tuning.html)
described online,
+There are [many more tuning
options](https://docs.oracle.com/en/java/javase/21/gctuning/introduction-garbage-collection-tuning.html)
described online,
but at a high level, managing how frequently full GC takes place can help in
reducing the overhead.
GC tuning flags for executors can be specified by setting
`spark.executor.defaultJavaOptions` or `spark.executor.extraJavaOptions` in
diff --git a/pom.xml b/pom.xml
index 9c644ee81dc3..9b16ddf3fba9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -121,8 +121,8 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
- <java.version>17</java.version>
- <java.minimum.version>17.0.11</java.minimum.version>
+ <java.version>21</java.version>
+ <java.minimum.version>21</java.minimum.version>
<maven.compiler.release>${java.version}</maven.compiler.release>
<maven.version>3.9.16</maven.version>
<exec-maven-plugin.version>3.6.1</exec-maven-plugin.version>
@@ -2836,7 +2836,7 @@
<arg>-feature</arg>
<arg>-explaintypes</arg>
<arg>-release</arg>
- <arg>17</arg>
+ <arg>21</arg>
<arg>-Wconf:any:e</arg>
<arg>-Wconf:cat=deprecation:wv</arg>
<arg>-Wunused:imports</arg>
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index 558ad51ac98f..27309d6120e7 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -216,7 +216,7 @@ Additional libraries that enhance functionality but are not
included in the inst
- **memory-profiler**: Used for PySpark UDF memory profiling,
``spark.profile.show(...)`` and ``spark.sql.pyspark.udf.profiler``.
- **plotly**: Used for PySpark plotting, ``DataFrame.plot``.
-Note that PySpark requires Java 17 or later with ``JAVA_HOME`` properly set
and refer to |downloading|_.
+Note that PySpark requires Java 21 or later with ``JAVA_HOME`` properly set
and refer to |downloading|_.
.. _optional-dependencies:
diff --git a/resource-managers/kubernetes/integration-tests/README.md
b/resource-managers/kubernetes/integration-tests/README.md
index f1373a26f05c..11a8c829feec 100644
--- a/resource-managers/kubernetes/integration-tests/README.md
+++ b/resource-managers/kubernetes/integration-tests/README.md
@@ -142,7 +142,7 @@ The tests are tagged, and the set of tests to run can be
controlled via the foll
## Customising the Build
-* `--java-version <version>` - Java version to build Spark and the tests with.
The default is `17`.
+* `--java-version <version>` - Java version to build Spark and the tests with.
The default is `21`.
* `--hadoop-profile <profile>` - Hadoop Maven profile to build with. The
default is `hadoop-3`.
* `--skip-building-dependencies` - skip rebuilding the modules the integration
tests depend on
(drops Maven's `-am` flag). Useful for iterating when the current tree is
already built.
diff --git
a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
index a0834e1ff23f..34c9f9e0ce76 100755
---
a/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
+++
b/resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh
@@ -38,7 +38,7 @@ CONTEXT=
INCLUDE_TAGS="k8s"
EXCLUDE_TAGS=
DEFAULT_EXCLUDE_TAGS="N/A"
-JAVA_VERSION="17"
+JAVA_VERSION="21"
BUILD_DEPENDENCIES_MVN_FLAG="-am"
HADOOP_PROFILE="hadoop-3"
MVN="$TEST_ROOT_DIR/build/mvn"
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
index d1eb63b2a942..624d5ed9fc72 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java
@@ -43,7 +43,7 @@ import org.apache.spark.unsafe.Platform;
* {@code org/apache/spark/sql/datasource/spark_datasource.h}, called through
the Foreign Function
* and Memory (FFM) API of Java 22 and later.
* <p>
- * Spark compiles for Java 17, where the FFM API does not exist, so the API is
only reached
+ * Spark compiles for Java 21, where the FFM API is still in preview, so the
API is only reached
* through method handles, looked up once when the first library is loaded.
The library exports
* {@code spark_ds_init}, which returns its table of functions, {@code
SparkDsLibrary}. Each
* function of the table is bound to a method handle that takes and returns
primitive values only,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]