This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3394c7b02846 [SPARK-60836][SQL][TEST] Support Spark versions like
`3.5.10` in `HiveExternalCatalogVersionsSuite`
3394c7b02846 is described below
commit 3394c7b02846710427e84c0ff2073d00b679f7f8
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 08:43:34 2026 -0800
[SPARK-60836][SQL][TEST] Support Spark versions like `3.5.10` in
`HiveExternalCatalogVersionsSuite`
### What changes were proposed in this pull request?
This PR aims to support Spark versions whose version components have
multiple digits, e.g., `3.5.10`, in `HiveExternalCatalogVersionsSuite`.
Specifically, `PROCESS_TABLES.testingVersions`
- matches `<a href="spark-(\d+\.\d+\.\d+)/">` instead of `<a
href="spark-(\d.\d.\d)/">` in the release listing.
- compares versions with `org.apache.spark.SPARK_VERSION` numerically via
`VersionUtils.majorMinorPatchVersion` instead of as strings.
### Why are the changes needed?
`branch-3.5` is `3.5.10-SNAPSHOT`, and Apache Spark 3.5.10 is likely to be
released in the next few months. Once `spark-3.5.10/` replaces `spark-3.5.9/`
in https://dist.apache.org/repos/dist/release/spark, this suite silently stops
testing Apache Spark 3.5 on Java 17 and still passes, because `(\d.\d.\d)`
matches a single digit per component and doesn't match `spark-3.5.10/`.
In addition, the string comparison with `SPARK_VERSION` is wrong for
multi-digit components. For example, `"3.5.9" < "3.5.10-SNAPSHOT"` is `false`
on `branch-3.5`.
This was pointed out in
https://github.com/apache/spark/pull/60049#discussion_r4451562553.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs. In addition, I ran this suite manually on Java 17 with two
release listings. This suite skips all old Spark versions on Java 21 and later.
- The real listing, which has `spark-3.5.9/`, `spark-4.0.5/`,
`spark-4.1.4/`, `spark-4.2.1/`, and `spark-4.3.0/`.
- A local copy of it in which `spark-3.5.10/` replaces `spark-3.5.9/`. To
skip the download, a copy of Apache Spark 3.5.9 is used as `spark-3.5.10` in
`spark.test.cache-dir`.
**Java 17 with `3.5.10` instead of `3.5.9`**
```
$ mkdir mirror
$ curl -sL https://dist.apache.org/repos/dist/release/spark | sed
's/spark-3\.5\.9/spark-3.5.10/g' > mirror/spark
$ cp -R $CACHE/spark-3.5.9 $CACHE/spark-3.5.10
$ SPARK_RELEASE_MIRROR=file://$PWD/mirror build/sbt -Phive
-Dspark.test.cache-dir=$CACHE 'hive/testOnly *HiveExternalCatalogVersionsSuite'
...
[info] HiveExternalCatalogVersionsSuite:
[info] - backward compatibility (14 seconds, 899 milliseconds)
...
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
| Java | Release listing | Old Spark versions that ran `spark-submit` |
`backward compatibility` |
| :-: | :-: | :-: | :-: |
| 17.0.21 | Real | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) |
| 17.0.21 | `3.5.10` instead of `3.5.9` | 3.5.10, 4.0.5, 4.1.4, 4.2.1,
4.3.0 | Passed (14 seconds) |
For `3.5.10`, the copied Apache Spark 3.5.9 ran `spark-submit`, and the
current build checked the tables of all five versions (`data_source_tbl_0` to
`data_source_tbl_4`). Without this PR, the second row runs only 4.0.5, 4.1.4,
4.2.1, and 4.3.0, and the current build checks four versions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60051 from dongjoon-hyun/SPARK-60836.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/HiveExternalCatalogVersionsSuite.scala | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index 46e0586eba3b..e3b35b9e6557 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -39,8 +39,8 @@ import org.apache.spark.sql.catalyst.catalog.CatalogTableType
import org.apache.spark.sql.classic.SparkSession
import org.apache.spark.sql.internal.StaticSQLConf.WAREHOUSE_PATH
import org.apache.spark.tags.{ExtendedHiveTest, SlowHiveTest}
+import org.apache.spark.util.{Utils, VersionUtils}
import org.apache.spark.util.ArrayImplicits._
-import org.apache.spark.util.Utils
/**
* Test HiveExternalCatalog backward compatibility.
@@ -272,15 +272,18 @@ object PROCESS_TABLES extends QueryTest {
val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
minimumTestingVersion.isDefined) {
import scala.io.Source
- val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
+ import scala.math.Ordering.Implicits._
+ val sparkVersionPattern = """<a href="spark-(\d+\.\d+\.\d+)/">""".r
+ // Compare versions numerically, e.g., 3.5.9 < 3.5.10, instead of as
strings.
+ def parseVersion(version: String) =
VersionUtils.majorMinorPatchVersion(version).get
try Utils.tryWithResource(
Source.fromURL(s"$releaseMirror/spark")) { source =>
source.mkString
.split("\n")
.filter(sparkVersionPattern.unanchored.matches(_))
.map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
- .filter(_ < org.apache.spark.SPARK_VERSION)
- .filter(v => minimumTestingVersion.exists(_ <= v))
+ .filter(v => parseVersion(v) <
parseVersion(org.apache.spark.SPARK_VERSION))
+ .filter(v => minimumTestingVersion.exists(m => parseVersion(m) <=
parseVersion(v)))
.filterNot(skipReleaseVersions.contains).toImmutableArraySeq
} catch {
// Do not throw exception during object initialization.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]