This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3394c7b02846 [SPARK-60836][SQL][TEST] Support Spark versions like 
`3.5.10` in `HiveExternalCatalogVersionsSuite`
3394c7b02846 is described below

commit 3394c7b02846710427e84c0ff2073d00b679f7f8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 08:43:34 2026 -0800

    [SPARK-60836][SQL][TEST] Support Spark versions like `3.5.10` in 
`HiveExternalCatalogVersionsSuite`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to support Spark versions whose version components have 
multiple digits, e.g., `3.5.10`, in `HiveExternalCatalogVersionsSuite`. 
Specifically, `PROCESS_TABLES.testingVersions`
    
    - matches `<a href="spark-(\d+\.\d+\.\d+)/">` instead of `<a 
href="spark-(\d.\d.\d)/">` in the release listing.
    - compares versions with `org.apache.spark.SPARK_VERSION` numerically via 
`VersionUtils.majorMinorPatchVersion` instead of as strings.
    
    ### Why are the changes needed?
    
    `branch-3.5` is `3.5.10-SNAPSHOT`, and Apache Spark 3.5.10 is likely to be 
released in the next few months. Once `spark-3.5.10/` replaces `spark-3.5.9/` 
in https://dist.apache.org/repos/dist/release/spark, this suite silently stops 
testing Apache Spark 3.5 on Java 17 and still passes, because `(\d.\d.\d)` 
matches a single digit per component and doesn't match `spark-3.5.10/`.
    
    In addition, the string comparison with `SPARK_VERSION` is wrong for 
multi-digit components. For example, `"3.5.9" < "3.5.10-SNAPSHOT"` is `false` 
on `branch-3.5`.
    
    This was pointed out in 
https://github.com/apache/spark/pull/60049#discussion_r4451562553.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a test-only change.
    
    ### How was this patch tested?
    
    Pass the CIs. In addition, I ran this suite manually on Java 17 with two 
release listings. This suite skips all old Spark versions on Java 21 and later.
    - The real listing, which has `spark-3.5.9/`, `spark-4.0.5/`, 
`spark-4.1.4/`, `spark-4.2.1/`, and `spark-4.3.0/`.
    - A local copy of it in which `spark-3.5.10/` replaces `spark-3.5.9/`. To 
skip the download, a copy of Apache Spark 3.5.9 is used as `spark-3.5.10` in 
`spark.test.cache-dir`.
    
    **Java 17 with `3.5.10` instead of `3.5.9`**
    ```
    $ mkdir mirror
    $ curl -sL https://dist.apache.org/repos/dist/release/spark | sed 
's/spark-3\.5\.9/spark-3.5.10/g' > mirror/spark
    $ cp -R $CACHE/spark-3.5.9 $CACHE/spark-3.5.10
    $ SPARK_RELEASE_MIRROR=file://$PWD/mirror build/sbt -Phive 
-Dspark.test.cache-dir=$CACHE 'hive/testOnly *HiveExternalCatalogVersionsSuite'
    ...
    [info] HiveExternalCatalogVersionsSuite:
    [info] - backward compatibility (14 seconds, 899 milliseconds)
    ...
    [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
    [info] All tests passed.
    ```
    
    | Java | Release listing | Old Spark versions that ran `spark-submit` | 
`backward compatibility` |
    | :-: | :-: | :-: | :-: |
    | 17.0.21 | Real | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) |
    | 17.0.21 | `3.5.10` instead of `3.5.9` | 3.5.10, 4.0.5, 4.1.4, 4.2.1, 
4.3.0 | Passed (14 seconds) |
    
    For `3.5.10`, the copied Apache Spark 3.5.9 ran `spark-submit`, and the 
current build checked the tables of all five versions (`data_source_tbl_0` to 
`data_source_tbl_4`). Without this PR, the second row runs only 4.0.5, 4.1.4, 
4.2.1, and 4.3.0, and the current build checks four versions.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60051 from dongjoon-hyun/SPARK-60836.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/HiveExternalCatalogVersionsSuite.scala     | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index 46e0586eba3b..e3b35b9e6557 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -39,8 +39,8 @@ import org.apache.spark.sql.catalyst.catalog.CatalogTableType
 import org.apache.spark.sql.classic.SparkSession
 import org.apache.spark.sql.internal.StaticSQLConf.WAREHOUSE_PATH
 import org.apache.spark.tags.{ExtendedHiveTest, SlowHiveTest}
+import org.apache.spark.util.{Utils, VersionUtils}
 import org.apache.spark.util.ArrayImplicits._
-import org.apache.spark.util.Utils
 
 /**
  * Test HiveExternalCatalog backward compatibility.
@@ -272,15 +272,18 @@ object PROCESS_TABLES extends QueryTest {
   val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
       minimumTestingVersion.isDefined) {
     import scala.io.Source
-    val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
+    import scala.math.Ordering.Implicits._
+    val sparkVersionPattern = """<a href="spark-(\d+\.\d+\.\d+)/">""".r
+    // Compare versions numerically, e.g., 3.5.9 < 3.5.10, instead of as 
strings.
+    def parseVersion(version: String) = 
VersionUtils.majorMinorPatchVersion(version).get
     try Utils.tryWithResource(
       Source.fromURL(s"$releaseMirror/spark")) { source =>
       source.mkString
         .split("\n")
         .filter(sparkVersionPattern.unanchored.matches(_))
         .map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
-        .filter(_ < org.apache.spark.SPARK_VERSION)
-        .filter(v => minimumTestingVersion.exists(_ <= v))
+        .filter(v => parseVersion(v) < 
parseVersion(org.apache.spark.SPARK_VERSION))
+        .filter(v => minimumTestingVersion.exists(m => parseVersion(m) <= 
parseVersion(v)))
         .filterNot(skipReleaseVersions.contains).toImmutableArraySeq
     } catch {
       // Do not throw exception during object initialization.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to