This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new ad920227c47c [SPARK-60833][SQL][TEST] Select old Spark versions by 
Java support in `HiveExternalCatalogVersionsSuite`
ad920227c47c is described below

commit ad920227c47ce772d148b78d84ed1834c2839984
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 08:28:03 2026 -0800

    [SPARK-60833][SQL][TEST] Select old Spark versions by Java support in 
`HiveExternalCatalogVersionsSuite`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to make `HiveExternalCatalogVersionsSuite` test the old Spark 
versions that support the running Java version, instead of skipping all of them 
on Java 21 and later.
    
    | Java | Release lines to test | Old Spark versions to test today |
    | :-: | :-: | :-: |
    | 17 | 3.3 and later (no change) | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 |
    | 21 | 4.0 and later | 4.0.5, 4.1.4, 4.2.1, 4.3.0 |
    | 25 | 4.2 and later | 4.2.1, 4.3.0 |
    | Others, e.g., 26 | None (no change) | None |
    
    Each minimum release line is the first release line whose documentation 
lists the Java version.
    - Apache Spark 3.3 to 3.5: `Spark runs on Java 8/11/17`
    - Apache Spark 4.0 and 4.1: `Spark runs on Java 17/21`
    - Apache Spark 4.2 and 4.3: `Spark runs on Java 17/21/25`
    
    No release line supports the other Java versions, e.g., Java 26. So, this 
suite keeps skipping the old Spark versions on them, and the log message shows 
the running Java version instead of `Java 21`.
    
    In addition, this PR uses `Runtime.version().feature()` instead of 
`Utils.isJavaVersionAtMost17`.
    
    ### Why are the changes needed?
    
    SPARK-44095 (Apache Spark 3.5.0) made this suite skip the old Spark 
versions on Java 21 because no Spark release supported Java 21 at that time. 
Now, Apache Spark 4.0.0 and later support Java 21, and Apache Spark 4.2.0 and 
later support Java 25. However, the Java 21 and Java 25 CIs still skip all old 
Spark versions.
    
    Moreover, SPARK-60669 (#59897) is going to drop Java 17 support at Apache 
Spark 5.0.0. After that, no `master` CI runs this suite on Java 17, and 
`master` would lose this backward compatibility test completely.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a test-only change.
    
    ### How was this patch tested?
    
    Pass the CIs. In addition, I ran this suite manually with Java 17, 21, 25, 
and 26.
    
    **Java 21**
    ```
    $ build/sbt -Phive 'hive/testOnly *HiveExternalCatalogVersionsSuite'
    ...
    [info] HiveExternalCatalogVersionsSuite:
    [info] - backward compatibility (16 seconds, 636 milliseconds)
    [info] Run completed in 52 seconds, 578 milliseconds.
    [info] Total number of tests run: 1
    [info] Suites: completed 1, aborted 0
    [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
    [info] All tests passed.
    
    $ grep -E 'SparkContext: (Running Spark version|Java version)' 
sql/hive/target/unit-tests.log | sed 's/.*SparkContext: //'
    Running Spark version 4.0.5
    Java version 21.0.13
    Running Spark version 4.1.4
    Java version 21.0.13+9-LTS
    Running Spark version 4.2.1
    Java version 21.0.13+9-LTS
    Running Spark version 4.3.0
    Java version 21.0.13+9-LTS
    ```
    
    **Summary**
    
    | Java | Old Spark versions that ran `spark-submit` | `backward 
compatibility` |
    | :-: | :-: | :-: |
    | 17.0.21 | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (17 seconds) |
    | 21.0.13 | 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) |
    | 25.0.5 | 4.2.1, 4.3.0 | Passed (13 seconds) |
    | 26.0.2 | None | Passed (11 milliseconds) |
    
    On Java 26, this suite logs `Skip tests because old Spark versions don't 
support Java 26.` and runs no old Spark version.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60049 from dongjoon-hyun/SPARK-60833.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 59bdfecc686d1fd0c8d967807a414881c37421d3)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/hive/HiveExternalCatalogVersionsSuite.scala      | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index d696dd06f391..46e0586eba3b 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -198,10 +198,11 @@ class HiveExternalCatalogVersionsSuite extends 
SparkSubmitTestUtils {
 
     if (PROCESS_TABLES.testingVersions.isEmpty) {
       if (PROCESS_TABLES.isPythonVersionAvailable) {
-        if (Utils.isJavaVersionAtMost17) {
+        if (PROCESS_TABLES.minimumTestingVersion.isDefined) {
           logError("Fail to get the latest Spark versions to test.")
         } else {
-          logInfo("Skip tests because old Spark versions don't support Java 
21.")
+          logInfo("Skip tests because old Spark versions don't support " +
+            s"Java ${PROCESS_TABLES.javaVersion}.")
         }
       } else {
         logError(s"Python version <  
${TestUtils.minimumPythonSupportedVersion}, " +
@@ -261,9 +262,15 @@ object PROCESS_TABLES extends QueryTest {
   val isPythonVersionAvailable = TestUtils.isPythonVersionAvailable
   val releaseMirror = sys.env.getOrElse("SPARK_RELEASE_MIRROR",
     "https://dist.apache.org/repos/dist/release";)
-  // Tests the latest version of every release line if Java version is at most 
17.
+  val javaVersion = Runtime.version().feature()
+  // Spark 3.3, 4.0, and 4.2 are the first release lines that support Java 17, 
21, and 25,
+  // respectively. No release line supports the other Java versions, e.g., 
Java 26. Add an entry
+  // here when a Spark release adds a Java version to its `docs/index.md`.
+  val minimumTestingVersion: Option[String] =
+    Map(17 -> "3.3.0", 21 -> "4.0.0", 25 -> "4.2.0").get(javaVersion)
+  // Tests the latest version of every release line that supports the running 
Java version.
   val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
-      Utils.isJavaVersionAtMost17) {
+      minimumTestingVersion.isDefined) {
     import scala.io.Source
     val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
     try Utils.tryWithResource(
@@ -273,6 +280,7 @@ object PROCESS_TABLES extends QueryTest {
         .filter(sparkVersionPattern.unanchored.matches(_))
         .map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
         .filter(_ < org.apache.spark.SPARK_VERSION)
+        .filter(v => minimumTestingVersion.exists(_ <= v))
         .filterNot(skipReleaseVersions.contains).toImmutableArraySeq
     } catch {
       // Do not throw exception during object initialization.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to