This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new da705b3f835c [SPARK-60896][CORE][SQL] Remove dead Java version
branches now that Java 21 is the minimum
da705b3f835c is described below
commit da705b3f835cd25005500e1c772fbaef727ffd2b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 19:35:35 2026 -0800
[SPARK-60896][CORE][SQL] Remove dead Java version branches now that Java 21
is the minimum
### What changes were proposed in this pull request?
Spark requires Java 21 or later since SPARK-60669 (`java.minimum.version`
is 21), so code that branched on the running Java version always takes the same
side. This PR removes the branches that can no longer run, and the helpers that
only served them:
- `SparkSystemUtils`: remove `isJavaVersionAtMost17` (always `false`).
`isJavaVersionAtLeast21` (always `true`) is kept for now because
`SQLQueryTestSuite`, `ThriftServerQueryTestSuite`, `IsolatedClientLoader` and
`HiveClientVersions` still use it in other open PRs; it can be removed once
they are merged.
- `RestSubmissionServer`: call
`Executors.newVirtualThreadPerTaskExecutor()` directly instead of the
`isJavaVersionAtLeast21` check plus the reflective lookup. The config
`spark.master.rest.virtualThread.enabled` keeps its name, default and behavior.
- Tests keep only the Java 21 side:
- `StandaloneRestSubmitSuite`: the virtual threads executor is always set.
- `UtilsSuite`: `createDirectory` on a symbolic link always succeeds
(drops the SPARK-50946 `17.0.14` branch).
- `QueryExecutionErrorsSuite`: the lambda class name is
`Outer$$Lambda/...` and the `substring` error message is `Range [5, 6) out of
bounds for length 5`.
- `WriteAheadLogSuite`: always use `ByteBuffer.allocate(0)` instead of
`mock[ByteBuffer]` (SPARK-40731), keeping a short comment on why.
- `Platform`: always pick the `DirectByteBuffer(long, long)` constructor.
`majorVersion` stays because the `< 26` cleaner selection still uses it.
- `ClosureCleaner`: drop the always-true `javaVersion >= 18` term in
`cloneIndyLambda`, and rename `getFinalModifiersFieldForJava17` to
`getFinalModifiersField` with its comments fixed (the logic is unchanged; the
fallback that uses it is only reached on Java 21 when
`jdk.reflect.useDirectMethodHandle` is not `true`).
The code that depends on Java 22, 24, 25 and 26 (`NativeLibrary`,
`SerializationDebugger`, `SizeEstimator`, `Platform`'s `< 26`,
`ClosureCleaner`'s `>= 22`) and `HiveExternalCatalogVersionsSuite`'s version
map are intentionally left as they are.
### Why are the changes needed?
After SPARK-60669, one side of each of these branches is dead code.
Removing it makes the code and the tests easier to read, and it avoids a
reflective call for an API that is always available.
### Does this PR introduce _any_ user-facing change?
No. Every removed branch was unreachable because the minimum Java version
is 21, so the remaining side is exactly what already ran.
### How was this patch tested?
Ran on JDK 21, and the suites below again on JDK 25:
```
build/sbt common-utils/Test/compile unsafe/Test/compile core/Test/compile
streaming/Test/compile sql/Test/compile
build/sbt "unsafe/test" \
"core/testOnly *StandaloneRestSubmitSuite
org.apache.spark.util.UtilsSuite *ClosureCleanerSuite*" \
"streaming/testOnly *WriteAheadLogSuite" "sql/testOnly
*QueryExecutionErrorsSuite"
```
- JDK 21: all pass (`unsafe` 161, `core` 125, `streaming` 31, `sql` 65
tests). JDK 25: all pass (`unsafe/testOnly *PlatformUtilSuite` 9, and the same
`core`, `streaming`, `sql` suites).
- `mimaReportBinaryIssues` for `common-utils` and `core` against 4.2.0
passes (`isJavaVersionAtMost17` is only on `private[spark]` `SparkSystemUtils`).
- `scalastyle` for `common-utils`, `core` (main and test), `streaming`
(test) and `sql` (test), and `checkstyle:check` for `common/unsafe` report no
violations.
- Also ran `Platform.allocateDirectBuffer` as a standalone smoke test on
JDK 21 and 25 with the new constructor selection.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 5.5
Closes #60111 from dongjoon-hyun/SPARK-60896.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../java/org/apache/spark/unsafe/Platform.java | 4 +---
.../org/apache/spark/util/ClosureCleaner.scala | 14 ++++++++------
.../org/apache/spark/util/SparkSystemUtils.scala | 5 -----
.../spark/deploy/rest/RestSubmissionServer.scala | 9 +++------
.../deploy/rest/StandaloneRestSubmitSuite.scala | 6 +-----
.../scala/org/apache/spark/util/UtilsSuite.scala | 9 +--------
.../sql/errors/QueryExecutionErrorsSuite.scala | 22 ++++------------------
.../spark/streaming/util/WriteAheadLogSuite.scala | 20 ++++++--------------
8 files changed, 24 insertions(+), 65 deletions(-)
diff --git a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
index efd9cc70d01c..df648c8805d2 100644
--- a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
+++ b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
@@ -65,9 +65,7 @@ public final class Platform {
try {
Class<?> cls = Class.forName("java.nio.DirectByteBuffer");
- Constructor<?> constructor = (majorVersion < 21) ?
- cls.getDeclaredConstructor(Long.TYPE, Integer.TYPE) :
- cls.getDeclaredConstructor(Long.TYPE, Long.TYPE);
+ Constructor<?> constructor = cls.getDeclaredConstructor(Long.TYPE,
Long.TYPE);
Field cleanerField = cls.getDeclaredField("cleaner");
if (!constructor.trySetAccessible()) {
constructor = null;
diff --git
a/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
b/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
index 36580123cead..dbf875820792 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
@@ -477,10 +477,12 @@ private[spark] object ClosureCleaner extends Logging {
parent = null, outerThis, capturingClass, accessedFields)
cloneIndyLambda(func, clonedOuterThis, lambdaProxy).getOrElse {
val outerField = func.getClass.getDeclaredField("arg$1")
- // SPARK-37072: When Java 17 is used and `outerField` is read-only,
+ // SPARK-37072: When `outerField` is read-only,
// the content of `outerField` cannot be set by reflect api directly.
// But we can remove the `final` modifier of `outerField` before set
value
// and reset the modifier after set value.
+ // This path is reached only on Java 21 when
`jdk.reflect.useDirectMethodHandle` is not
+ // `true`, e.g., with `-Djdk.reflect.useDirectMethodHandle=false`.
setFieldAndIgnoreModifiers(func, outerField, clonedOuterThis)
func
}
@@ -616,7 +618,7 @@ private[spark] object ClosureCleaner extends Logging {
}
private def setFieldAndIgnoreModifiers(obj: AnyRef, field: Field, value:
AnyRef): Unit = {
- val modifiersField = getFinalModifiersFieldForJava17(field)
+ val modifiersField = getFinalModifiersField(field)
modifiersField
.foreach(m => m.setInt(field, field.getModifiers & ~Modifier.FINAL))
field.setAccessible(true)
@@ -627,9 +629,10 @@ private[spark] object ClosureCleaner extends Logging {
}
/**
- * This method is used to get the final modifier field when on Java 17.
+ * This method is used to get the `modifiers` field of
`java.lang.reflect.Field`
+ * when the given `field` is final, so that the `final` modifier can be
removed temporarily.
*/
- private def getFinalModifiersFieldForJava17(field: Field): Option[Field] = {
+ private def getFinalModifiersField(field: Field): Option[Field] = {
if (Modifier.isFinal(field.getModifiers)) {
val methodGetDeclaredFields0 = classOf[Class[_]]
.getDeclaredMethod("getDeclaredFields0", classOf[Boolean])
@@ -664,8 +667,7 @@ private[spark] object ClosureCleaner extends Logging {
val javaVersion = Runtime.version().feature()
val useClone =
System.getProperty("spark.cloneBasedClosureCleaner.enabled") == "true" ||
System.getenv("SPARK_CLONE_BASED_CLOSURE_CLEANER") == "1" || javaVersion
>= 22 ||
- (javaVersion >= 18 &&
- System.getProperty("jdk.reflect.useDirectMethodHandle", "true") ==
"true")
+ System.getProperty("jdk.reflect.useDirectMethodHandle", "true") == "true"
if (useClone) {
val factory = makeClonedIndyLambdaFactory(indyLambda.getClass,
lambdaProxy)
diff --git
a/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
b/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
index 72c62c20d16e..17f6296831fe 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
@@ -39,11 +39,6 @@ private[spark] trait SparkSystemUtils {
*/
val javaVersion = JavaUtils.javaVersion
- /**
- * Whether the underlying Java version is at most 17.
- */
- val isJavaVersionAtMost17 = Runtime.version().feature() <= 17
-
/**
* Whether the underlying Java version is at least 21.
*/
diff --git
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
index 88d98d0236a6..91cbbe45a447 100644
---
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
@@ -19,7 +19,7 @@ package org.apache.spark.deploy.rest
import java.nio.charset.StandardCharsets
import java.util.EnumSet
-import java.util.concurrent.{Executors, ExecutorService}
+import java.util.concurrent.Executors
import com.fasterxml.jackson.core.JsonProcessingException
import jakarta.servlet.DispatcherType
@@ -99,11 +99,8 @@ private[spark] abstract class RestSubmissionServer(
private def doStart(startPort: Int): (Server, Int) = {
val threadPool = new
QueuedThreadPool(masterConf.get(MASTER_REST_SERVER_MAX_THREADS))
threadPool.setName(getClass().getSimpleName())
- if (Utils.isJavaVersionAtLeast21 &&
masterConf.get(MASTER_REST_SERVER_VIRTUAL_THREADS)) {
- val newVirtualThreadPerTaskExecutor =
- classOf[Executors].getMethod("newVirtualThreadPerTaskExecutor")
- val service =
newVirtualThreadPerTaskExecutor.invoke(null).asInstanceOf[ExecutorService]
- threadPool.setVirtualThreadsExecutor(service)
+ if (masterConf.get(MASTER_REST_SERVER_VIRTUAL_THREADS)) {
+
threadPool.setVirtualThreadsExecutor(Executors.newVirtualThreadPerTaskExecutor())
}
threadPool.setDaemon(true)
val server = new Server(threadPool)
diff --git
a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
index 5ab6503afadb..c37812e5a1c7 100644
---
a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
+++
b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
@@ -643,11 +643,7 @@ class StandaloneRestSubmitSuite extends SparkFunSuite {
server = Some(new StandaloneRestServer(localhost, 0, conf, fakeMasterRef,
"spark://fake:7077"))
server.get.start()
val pool =
server.get._server.get.getThreadPool.asInstanceOf[QueuedThreadPool]
- if (Utils.isJavaVersionAtLeast21) {
- assert(pool.getVirtualThreadsExecutor != null)
- } else {
- assert(pool.getVirtualThreadsExecutor == null)
- }
+ assert(pool.getVirtualThreadsExecutor != null)
}
test("SPARK-58049: Reject an over-sized request body with
SC_REQUEST_ENTITY_TOO_LARGE") {
diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
index cf68516b6802..a76c822d02ad 100644
--- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
@@ -552,14 +552,7 @@ class UtilsSuite extends SparkFunSuite with
ResetSystemProperties {
// 6. Symbolic link
val scenario6 = Files.createSymbolicLink(new File(testDir, "scenario6")
.toPath, scenario1.toPath).toFile
- if (Utils.isJavaVersionAtLeast21) {
- assert(Utils.createDirectory(scenario6))
- } else if (Runtime.version().feature() == 17 && Runtime.version().update()
>= 14) {
- // SPARK-50946: Java 17.0.14 includes JDK-8294193, so scenario6 can
succeed.
- assert(Utils.createDirectory(scenario6))
- } else {
- assert(!Utils.createDirectory(scenario6))
- }
+ assert(Utils.createDirectory(scenario6))
assert(scenario6.exists())
// 7. Directory exists
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
index 218c74c6efcd..1d2f404211bd 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
@@ -526,16 +526,9 @@ class QueryExecutionErrorsSuite
}}
spark.udf.register("luckyCharOfWord", luckyCharOfWord)
- val functionNameRegex = if (Utils.isJavaVersionAtLeast21) {
- "`luckyCharOfWord \\(QueryExecutionErrorsSuite\\$\\$Lambda/\\w+\\)`"
- } else {
- "`luckyCharOfWord
\\(QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+\\)`"
- }
- val reason = if (Utils.isJavaVersionAtLeast21) {
+ val functionNameRegex = "`luckyCharOfWord
\\(QueryExecutionErrorsSuite\\$\\$Lambda/\\w+\\)`"
+ val reason =
"java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of
bounds for length 5"
- } else {
- "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
- }
checkError(
exception = intercept[SparkException] {
@@ -557,16 +550,9 @@ class QueryExecutionErrorsSuite
val luckyCharOfWord = udf { (word: String, index: Int) => {
word.substring(index, index + 1)
}}
- val functionNameRegex = if (Utils.isJavaVersionAtLeast21) {
- "`QueryExecutionErrorsSuite\\$\\$Lambda/\\w+`"
- } else {
- "`QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+`"
- }
- val reason = if (Utils.isJavaVersionAtLeast21) {
+ val functionNameRegex = "`QueryExecutionErrorsSuite\\$\\$Lambda/\\w+`"
+ val reason =
"java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of
bounds for length 5"
- } else {
- "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
- }
checkError(
exception = intercept[SparkException] {
diff --git
a/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
b/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
index 81e19a51b327..3543198988e7 100644
---
a/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
+++
b/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
@@ -476,13 +476,9 @@ class BatchedWriteAheadLogSuite extends
CommonWriteAheadLogTests(
val batchedWal = new BatchedWriteAheadLog(wal, sparkConf)
val e = intercept[SparkException] {
- val buffer = if (Utils.isJavaVersionAtMost17) {
- mock[ByteBuffer]
- } else {
- // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
- // for Java 17+ due to mockito 4 can't mock/spy sealed class
- ByteBuffer.allocate(0)
- }
+ // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
+ // because mockito can't mock/spy a sealed class
+ val buffer = ByteBuffer.allocate(0)
batchedWal.write(buffer, 2L)
}
assert(e.getCause.getMessage === "Hello!")
@@ -552,13 +548,9 @@ class BatchedWriteAheadLogSuite extends
CommonWriteAheadLogTests(
batchedWal.close()
verify(wal, times(1)).close()
- val buffer = if (Utils.isJavaVersionAtMost17) {
- mock[ByteBuffer]
- } else {
- // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
- // for Java 17+ due to mockito 4 can't mock/spy sealed class
- ByteBuffer.allocate(0)
- }
+ // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
+ // because mockito can't mock/spy a sealed class
+ val buffer = ByteBuffer.allocate(0)
intercept[IllegalStateException](batchedWal.write(buffer, 12L))
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]