This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new da705b3f835c [SPARK-60896][CORE][SQL] Remove dead Java version 
branches now that Java 21 is the minimum
da705b3f835c is described below

commit da705b3f835cd25005500e1c772fbaef727ffd2b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 19:35:35 2026 -0800

    [SPARK-60896][CORE][SQL] Remove dead Java version branches now that Java 21 
is the minimum
    
    ### What changes were proposed in this pull request?
    
    Spark requires Java 21 or later since SPARK-60669 (`java.minimum.version` 
is 21), so code that branched on the running Java version always takes the same 
side. This PR removes the branches that can no longer run, and the helpers that 
only served them:
    
    - `SparkSystemUtils`: remove `isJavaVersionAtMost17` (always `false`). 
`isJavaVersionAtLeast21` (always `true`) is kept for now because 
`SQLQueryTestSuite`, `ThriftServerQueryTestSuite`, `IsolatedClientLoader` and 
`HiveClientVersions` still use it in other open PRs; it can be removed once 
they are merged.
    - `RestSubmissionServer`: call 
`Executors.newVirtualThreadPerTaskExecutor()` directly instead of the 
`isJavaVersionAtLeast21` check plus the reflective lookup. The config 
`spark.master.rest.virtualThread.enabled` keeps its name, default and behavior.
    - Tests keep only the Java 21 side:
      - `StandaloneRestSubmitSuite`: the virtual threads executor is always set.
      - `UtilsSuite`: `createDirectory` on a symbolic link always succeeds 
(drops the SPARK-50946 `17.0.14` branch).
      - `QueryExecutionErrorsSuite`: the lambda class name is 
`Outer$$Lambda/...` and the `substring` error message is `Range [5, 6) out of 
bounds for length 5`.
      - `WriteAheadLogSuite`: always use `ByteBuffer.allocate(0)` instead of 
`mock[ByteBuffer]` (SPARK-40731), keeping a short comment on why.
    - `Platform`: always pick the `DirectByteBuffer(long, long)` constructor. 
`majorVersion` stays because the `< 26` cleaner selection still uses it.
    - `ClosureCleaner`: drop the always-true `javaVersion >= 18` term in 
`cloneIndyLambda`, and rename `getFinalModifiersFieldForJava17` to 
`getFinalModifiersField` with its comments fixed (the logic is unchanged; the 
fallback that uses it is only reached on Java 21 when 
`jdk.reflect.useDirectMethodHandle` is not `true`).
    
    The code that depends on Java 22, 24, 25 and 26 (`NativeLibrary`, 
`SerializationDebugger`, `SizeEstimator`, `Platform`'s `< 26`, 
`ClosureCleaner`'s `>= 22`) and `HiveExternalCatalogVersionsSuite`'s version 
map are intentionally left as they are.
    
    ### Why are the changes needed?
    
    After SPARK-60669, one side of each of these branches is dead code. 
Removing it makes the code and the tests easier to read, and it avoids a 
reflective call for an API that is always available.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Every removed branch was unreachable because the minimum Java version 
is 21, so the remaining side is exactly what already ran.
    
    ### How was this patch tested?
    
    Ran on JDK 21, and the suites below again on JDK 25:
    
    ```
    build/sbt common-utils/Test/compile unsafe/Test/compile core/Test/compile 
streaming/Test/compile sql/Test/compile
    build/sbt "unsafe/test" \
      "core/testOnly *StandaloneRestSubmitSuite 
org.apache.spark.util.UtilsSuite *ClosureCleanerSuite*" \
      "streaming/testOnly *WriteAheadLogSuite" "sql/testOnly 
*QueryExecutionErrorsSuite"
    ```
    
    - JDK 21: all pass (`unsafe` 161, `core` 125, `streaming` 31, `sql` 65 
tests). JDK 25: all pass (`unsafe/testOnly *PlatformUtilSuite` 9, and the same 
`core`, `streaming`, `sql` suites).
    - `mimaReportBinaryIssues` for `common-utils` and `core` against 4.2.0 
passes (`isJavaVersionAtMost17` is only on `private[spark]` `SparkSystemUtils`).
    - `scalastyle` for `common-utils`, `core` (main and test), `streaming` 
(test) and `sql` (test), and `checkstyle:check` for `common/unsafe` report no 
violations.
    - Also ran `Platform.allocateDirectBuffer` as a standalone smoke test on 
JDK 21 and 25 with the new constructor selection.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Sonnet 5.5
    
    Closes #60111 from dongjoon-hyun/SPARK-60896.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../java/org/apache/spark/unsafe/Platform.java     |  4 +---
 .../org/apache/spark/util/ClosureCleaner.scala     | 14 ++++++++------
 .../org/apache/spark/util/SparkSystemUtils.scala   |  5 -----
 .../spark/deploy/rest/RestSubmissionServer.scala   |  9 +++------
 .../deploy/rest/StandaloneRestSubmitSuite.scala    |  6 +-----
 .../scala/org/apache/spark/util/UtilsSuite.scala   |  9 +--------
 .../sql/errors/QueryExecutionErrorsSuite.scala     | 22 ++++------------------
 .../spark/streaming/util/WriteAheadLogSuite.scala  | 20 ++++++--------------
 8 files changed, 24 insertions(+), 65 deletions(-)

diff --git a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java 
b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
index efd9cc70d01c..df648c8805d2 100644
--- a/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
+++ b/common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
@@ -65,9 +65,7 @@ public final class Platform {
 
     try {
       Class<?> cls = Class.forName("java.nio.DirectByteBuffer");
-      Constructor<?> constructor = (majorVersion < 21) ?
-        cls.getDeclaredConstructor(Long.TYPE, Integer.TYPE) :
-        cls.getDeclaredConstructor(Long.TYPE, Long.TYPE);
+      Constructor<?> constructor = cls.getDeclaredConstructor(Long.TYPE, 
Long.TYPE);
       Field cleanerField = cls.getDeclaredField("cleaner");
       if (!constructor.trySetAccessible()) {
         constructor = null;
diff --git 
a/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala 
b/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
index 36580123cead..dbf875820792 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala
@@ -477,10 +477,12 @@ private[spark] object ClosureCleaner extends Logging {
         parent = null, outerThis, capturingClass, accessedFields)
       cloneIndyLambda(func, clonedOuterThis, lambdaProxy).getOrElse {
         val outerField = func.getClass.getDeclaredField("arg$1")
-        // SPARK-37072: When Java 17 is used and `outerField` is read-only,
+        // SPARK-37072: When `outerField` is read-only,
         // the content of `outerField` cannot be set by reflect api directly.
         // But we can remove the `final` modifier of `outerField` before set 
value
         // and reset the modifier after set value.
+        // This path is reached only on Java 21 when 
`jdk.reflect.useDirectMethodHandle` is not
+        // `true`, e.g., with `-Djdk.reflect.useDirectMethodHandle=false`.
         setFieldAndIgnoreModifiers(func, outerField, clonedOuterThis)
         func
       }
@@ -616,7 +618,7 @@ private[spark] object ClosureCleaner extends Logging {
   }
 
   private def setFieldAndIgnoreModifiers(obj: AnyRef, field: Field, value: 
AnyRef): Unit = {
-    val modifiersField = getFinalModifiersFieldForJava17(field)
+    val modifiersField = getFinalModifiersField(field)
     modifiersField
       .foreach(m => m.setInt(field, field.getModifiers & ~Modifier.FINAL))
     field.setAccessible(true)
@@ -627,9 +629,10 @@ private[spark] object ClosureCleaner extends Logging {
   }
 
   /**
-   * This method is used to get the final modifier field when on Java 17.
+   * This method is used to get the `modifiers` field of 
`java.lang.reflect.Field`
+   * when the given `field` is final, so that the `final` modifier can be 
removed temporarily.
    */
-  private def getFinalModifiersFieldForJava17(field: Field): Option[Field] = {
+  private def getFinalModifiersField(field: Field): Option[Field] = {
     if (Modifier.isFinal(field.getModifiers)) {
       val methodGetDeclaredFields0 = classOf[Class[_]]
         .getDeclaredMethod("getDeclaredFields0", classOf[Boolean])
@@ -664,8 +667,7 @@ private[spark] object ClosureCleaner extends Logging {
     val javaVersion = Runtime.version().feature()
     val useClone = 
System.getProperty("spark.cloneBasedClosureCleaner.enabled") == "true" ||
       System.getenv("SPARK_CLONE_BASED_CLOSURE_CLEANER") == "1" || javaVersion 
>= 22 ||
-      (javaVersion >= 18 &&
-        System.getProperty("jdk.reflect.useDirectMethodHandle", "true") == 
"true")
+      System.getProperty("jdk.reflect.useDirectMethodHandle", "true") == "true"
 
     if (useClone) {
       val factory = makeClonedIndyLambdaFactory(indyLambda.getClass, 
lambdaProxy)
diff --git 
a/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala 
b/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
index 72c62c20d16e..17f6296831fe 100644
--- a/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
+++ b/common/utils/src/main/scala/org/apache/spark/util/SparkSystemUtils.scala
@@ -39,11 +39,6 @@ private[spark] trait SparkSystemUtils {
    */
   val javaVersion = JavaUtils.javaVersion
 
-  /**
-   * Whether the underlying Java version is at most 17.
-   */
-  val isJavaVersionAtMost17 = Runtime.version().feature() <= 17
-
   /**
    * Whether the underlying Java version is at least 21.
    */
diff --git 
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala 
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
index 88d98d0236a6..91cbbe45a447 100644
--- 
a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
+++ 
b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala
@@ -19,7 +19,7 @@ package org.apache.spark.deploy.rest
 
 import java.nio.charset.StandardCharsets
 import java.util.EnumSet
-import java.util.concurrent.{Executors, ExecutorService}
+import java.util.concurrent.Executors
 
 import com.fasterxml.jackson.core.JsonProcessingException
 import jakarta.servlet.DispatcherType
@@ -99,11 +99,8 @@ private[spark] abstract class RestSubmissionServer(
   private def doStart(startPort: Int): (Server, Int) = {
     val threadPool = new 
QueuedThreadPool(masterConf.get(MASTER_REST_SERVER_MAX_THREADS))
     threadPool.setName(getClass().getSimpleName())
-    if (Utils.isJavaVersionAtLeast21 && 
masterConf.get(MASTER_REST_SERVER_VIRTUAL_THREADS)) {
-      val newVirtualThreadPerTaskExecutor =
-        classOf[Executors].getMethod("newVirtualThreadPerTaskExecutor")
-      val service = 
newVirtualThreadPerTaskExecutor.invoke(null).asInstanceOf[ExecutorService]
-      threadPool.setVirtualThreadsExecutor(service)
+    if (masterConf.get(MASTER_REST_SERVER_VIRTUAL_THREADS)) {
+      
threadPool.setVirtualThreadsExecutor(Executors.newVirtualThreadPerTaskExecutor())
     }
     threadPool.setDaemon(true)
     val server = new Server(threadPool)
diff --git 
a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
 
b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
index 5ab6503afadb..c37812e5a1c7 100644
--- 
a/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
+++ 
b/core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala
@@ -643,11 +643,7 @@ class StandaloneRestSubmitSuite extends SparkFunSuite {
     server = Some(new StandaloneRestServer(localhost, 0, conf, fakeMasterRef, 
"spark://fake:7077"))
     server.get.start()
     val pool = 
server.get._server.get.getThreadPool.asInstanceOf[QueuedThreadPool]
-    if (Utils.isJavaVersionAtLeast21) {
-      assert(pool.getVirtualThreadsExecutor != null)
-    } else {
-      assert(pool.getVirtualThreadsExecutor == null)
-    }
+    assert(pool.getVirtualThreadsExecutor != null)
   }
 
   test("SPARK-58049: Reject an over-sized request body with 
SC_REQUEST_ENTITY_TOO_LARGE") {
diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala 
b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
index cf68516b6802..a76c822d02ad 100644
--- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
@@ -552,14 +552,7 @@ class UtilsSuite extends SparkFunSuite with 
ResetSystemProperties {
     // 6. Symbolic link
     val scenario6 = Files.createSymbolicLink(new File(testDir, "scenario6")
       .toPath, scenario1.toPath).toFile
-    if (Utils.isJavaVersionAtLeast21) {
-      assert(Utils.createDirectory(scenario6))
-    } else if (Runtime.version().feature() == 17 && Runtime.version().update() 
>= 14) {
-      // SPARK-50946: Java 17.0.14 includes JDK-8294193, so scenario6 can 
succeed.
-      assert(Utils.createDirectory(scenario6))
-    } else {
-      assert(!Utils.createDirectory(scenario6))
-    }
+    assert(Utils.createDirectory(scenario6))
     assert(scenario6.exists())
 
     // 7. Directory exists
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
index 218c74c6efcd..1d2f404211bd 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
@@ -526,16 +526,9 @@ class QueryExecutionErrorsSuite
     }}
     spark.udf.register("luckyCharOfWord", luckyCharOfWord)
 
-    val functionNameRegex = if (Utils.isJavaVersionAtLeast21) {
-      "`luckyCharOfWord \\(QueryExecutionErrorsSuite\\$\\$Lambda/\\w+\\)`"
-    } else {
-      "`luckyCharOfWord 
\\(QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+\\)`"
-    }
-    val reason = if (Utils.isJavaVersionAtLeast21) {
+    val functionNameRegex = "`luckyCharOfWord 
\\(QueryExecutionErrorsSuite\\$\\$Lambda/\\w+\\)`"
+    val reason =
       "java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of 
bounds for length 5"
-    } else {
-      "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
-    }
 
     checkError(
       exception = intercept[SparkException] {
@@ -557,16 +550,9 @@ class QueryExecutionErrorsSuite
     val luckyCharOfWord = udf { (word: String, index: Int) => {
       word.substring(index, index + 1)
     }}
-    val functionNameRegex = if (Utils.isJavaVersionAtLeast21) {
-      "`QueryExecutionErrorsSuite\\$\\$Lambda/\\w+`"
-    } else {
-      "`QueryExecutionErrorsSuite\\$\\$Lambda\\$\\d+/\\w+`"
-    }
-    val reason = if (Utils.isJavaVersionAtLeast21) {
+    val functionNameRegex = "`QueryExecutionErrorsSuite\\$\\$Lambda/\\w+`"
+    val reason =
       "java.lang.StringIndexOutOfBoundsException: Range \\[5, 6\\) out of 
bounds for length 5"
-    } else {
-      "java.lang.StringIndexOutOfBoundsException: begin 5, end 6, length 5"
-    }
 
     checkError(
       exception = intercept[SparkException] {
diff --git 
a/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
 
b/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
index 81e19a51b327..3543198988e7 100644
--- 
a/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
+++ 
b/streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala
@@ -476,13 +476,9 @@ class BatchedWriteAheadLogSuite extends 
CommonWriteAheadLogTests(
     val batchedWal = new BatchedWriteAheadLog(wal, sparkConf)
 
     val e = intercept[SparkException] {
-      val buffer = if (Utils.isJavaVersionAtMost17) {
-        mock[ByteBuffer]
-      } else {
-        // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
-        // for Java 17+ due to mockito 4 can't mock/spy sealed class
-        ByteBuffer.allocate(0)
-      }
+      // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
+      // because mockito can't mock/spy a sealed class
+      val buffer = ByteBuffer.allocate(0)
       batchedWal.write(buffer, 2L)
     }
     assert(e.getCause.getMessage === "Hello!")
@@ -552,13 +548,9 @@ class BatchedWriteAheadLogSuite extends 
CommonWriteAheadLogTests(
     batchedWal.close()
     verify(wal, times(1)).close()
 
-    val buffer = if (Utils.isJavaVersionAtMost17) {
-      mock[ByteBuffer]
-    } else {
-      // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
-      // for Java 17+ due to mockito 4 can't mock/spy sealed class
-      ByteBuffer.allocate(0)
-    }
+    // SPARK-40731: Use a 0 size `ByteBuffer` instead of `mock[ByteBuffer]`
+    // because mockito can't mock/spy a sealed class
+    val buffer = ByteBuffer.allocate(0)
     intercept[IllegalStateException](batchedWal.write(buffer, 12L))
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to