Hi, All. I'd like to propose dropping Java 17 support in Apache Spark 5.0.0, making Java 21 the minimum supported Java version. This is the same step we took in Apache Spark 4.0.0 for Java 8 and 11 (SPARK-44112).
https://issues.apache.org/jira/browse/SPARK-60669 https://github.com/apache/spark/pull/59897 Here are the reasons. 1. Java 21 is mature in our community. Apache Spark has officially supported Java 21 since 4.0.0 (May 2025), and our daily CI has tested Java 21 since June 2023 (SPARK-43835). 2. Supporting both Java 17 and 21 has a maintenance cost. Since the PR builder uses Java 17, we keep separate Java 21 SQL golden files (`*.sql.out.java21`). They have become stale after merges and had to be regenerated separately several times (SPARK-51937, SPARK-52400, SPARK-57560, SPARK-57649). We also keep Java version-specific code paths and tests such as `Utils.isJavaVersionAtLeast21`. 3. The ecosystem is moving on. For example, our Hive 4.2 metastore support (SPARK-57263) requires Java 21, and newer dependency versions such as Apache Derby 10.17 no longer support Java 17. 4. We can use Java 21 APIs and language features directly, e.g., virtual threads without reflection. This affects only the `master` branch (Apache Spark 5.0.0). `branch-4.x` (Apache Spark 4.4.0) and the other 4.x branches keep Java 17 support, so Java 17 users can stay on the Apache Spark 4.x line. Apache Spark 5.0.0 will support Java 21 and 25. SPARK-60669 changes the minimum Java version in the build, the default Java version in GitHub Actions, and the documentation. I'll clean up the remaining Java 17-specific code, tests, CI images, and release scripts in separate PRs. Please let me know if you have any concerns. Thanks, Dongjoon Hyun. --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
