This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 59bdfecc686d [SPARK-60833][SQL][TEST] Select old Spark versions by
Java support in `HiveExternalCatalogVersionsSuite`
59bdfecc686d is described below
commit 59bdfecc686d1fd0c8d967807a414881c37421d3
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 08:28:03 2026 -0800
[SPARK-60833][SQL][TEST] Select old Spark versions by Java support in
`HiveExternalCatalogVersionsSuite`
### What changes were proposed in this pull request?
This PR aims to make `HiveExternalCatalogVersionsSuite` test the old Spark
versions that support the running Java version, instead of skipping all of them
on Java 21 and later.
| Java | Release lines to test | Old Spark versions to test today |
| :-: | :-: | :-: |
| 17 | 3.3 and later (no change) | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 |
| 21 | 4.0 and later | 4.0.5, 4.1.4, 4.2.1, 4.3.0 |
| 25 | 4.2 and later | 4.2.1, 4.3.0 |
| Others, e.g., 26 | None (no change) | None |
Each minimum release line is the first release line whose documentation
lists the Java version.
- Apache Spark 3.3 to 3.5: `Spark runs on Java 8/11/17`
- Apache Spark 4.0 and 4.1: `Spark runs on Java 17/21`
- Apache Spark 4.2 and 4.3: `Spark runs on Java 17/21/25`
No release line supports the other Java versions, e.g., Java 26. So, this
suite keeps skipping the old Spark versions on them, and the log message shows
the running Java version instead of `Java 21`.
In addition, this PR uses `Runtime.version().feature()` instead of
`Utils.isJavaVersionAtMost17`.
### Why are the changes needed?
SPARK-44095 (Apache Spark 3.5.0) made this suite skip the old Spark
versions on Java 21 because no Spark release supported Java 21 at that time.
Now, Apache Spark 4.0.0 and later support Java 21, and Apache Spark 4.2.0 and
later support Java 25. However, the Java 21 and Java 25 CIs still skip all old
Spark versions.
Moreover, SPARK-60669 (#59897) is going to drop Java 17 support at Apache
Spark 5.0.0. After that, no `master` CI runs this suite on Java 17, and
`master` would lose this backward compatibility test completely.
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs. In addition, I ran this suite manually with Java 17, 21, 25,
and 26.
**Java 21**
```
$ build/sbt -Phive 'hive/testOnly *HiveExternalCatalogVersionsSuite'
...
[info] HiveExternalCatalogVersionsSuite:
[info] - backward compatibility (16 seconds, 636 milliseconds)
[info] Run completed in 52 seconds, 578 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
$ grep -E 'SparkContext: (Running Spark version|Java version)'
sql/hive/target/unit-tests.log | sed 's/.*SparkContext: //'
Running Spark version 4.0.5
Java version 21.0.13
Running Spark version 4.1.4
Java version 21.0.13+9-LTS
Running Spark version 4.2.1
Java version 21.0.13+9-LTS
Running Spark version 4.3.0
Java version 21.0.13+9-LTS
```
**Summary**
| Java | Old Spark versions that ran `spark-submit` | `backward
compatibility` |
| :-: | :-: | :-: |
| 17.0.21 | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (17 seconds) |
| 21.0.13 | 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) |
| 25.0.5 | 4.2.1, 4.3.0 | Passed (13 seconds) |
| 26.0.2 | None | Passed (11 milliseconds) |
On Java 26, this suite logs `Skip tests because old Spark versions don't
support Java 26.` and runs no old Spark version.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60049 from dongjoon-hyun/SPARK-60833.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/hive/HiveExternalCatalogVersionsSuite.scala | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
index d696dd06f391..46e0586eba3b 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
@@ -198,10 +198,11 @@ class HiveExternalCatalogVersionsSuite extends
SparkSubmitTestUtils {
if (PROCESS_TABLES.testingVersions.isEmpty) {
if (PROCESS_TABLES.isPythonVersionAvailable) {
- if (Utils.isJavaVersionAtMost17) {
+ if (PROCESS_TABLES.minimumTestingVersion.isDefined) {
logError("Fail to get the latest Spark versions to test.")
} else {
- logInfo("Skip tests because old Spark versions don't support Java
21.")
+ logInfo("Skip tests because old Spark versions don't support " +
+ s"Java ${PROCESS_TABLES.javaVersion}.")
}
} else {
logError(s"Python version <
${TestUtils.minimumPythonSupportedVersion}, " +
@@ -261,9 +262,15 @@ object PROCESS_TABLES extends QueryTest {
val isPythonVersionAvailable = TestUtils.isPythonVersionAvailable
val releaseMirror = sys.env.getOrElse("SPARK_RELEASE_MIRROR",
"https://dist.apache.org/repos/dist/release")
- // Tests the latest version of every release line if Java version is at most
17.
+ val javaVersion = Runtime.version().feature()
+ // Spark 3.3, 4.0, and 4.2 are the first release lines that support Java 17,
21, and 25,
+ // respectively. No release line supports the other Java versions, e.g.,
Java 26. Add an entry
+ // here when a Spark release adds a Java version to its `docs/index.md`.
+ val minimumTestingVersion: Option[String] =
+ Map(17 -> "3.3.0", 21 -> "4.0.0", 25 -> "4.2.0").get(javaVersion)
+ // Tests the latest version of every release line that supports the running
Java version.
val testingVersions: Seq[String] = if (isPythonVersionAvailable &&
- Utils.isJavaVersionAtMost17) {
+ minimumTestingVersion.isDefined) {
import scala.io.Source
val sparkVersionPattern = """<a href="spark-(\d.\d.\d)/">""".r
try Utils.tryWithResource(
@@ -273,6 +280,7 @@ object PROCESS_TABLES extends QueryTest {
.filter(sparkVersionPattern.unanchored.matches(_))
.map(sparkVersionPattern.findFirstMatchIn(_).get.group(1))
.filter(_ < org.apache.spark.SPARK_VERSION)
+ .filter(v => minimumTestingVersion.exists(_ <= v))
.filterNot(skipReleaseVersions.contains).toImmutableArraySeq
} catch {
// Do not throw exception during object initialization.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]