dongjoon-hyun opened a new pull request, #60051: URL: https://github.com/apache/spark/pull/60051
### What changes were proposed in this pull request? This PR aims to support Spark versions whose version components have multiple digits, e.g., `3.5.10`, in `HiveExternalCatalogVersionsSuite`. Specifically, `PROCESS_TABLES.testingVersions` - matches `<a href="spark-(\d+\.\d+\.\d+)/">` instead of `<a href="spark-(\d.\d.\d)/">` in the release listing. - compares versions with `org.apache.spark.SPARK_VERSION` numerically via `VersionUtils.majorMinorPatchVersion` instead of as strings. ### Why are the changes needed? `branch-3.5` is `3.5.10-SNAPSHOT`, and Apache Spark 3.5.10 is likely to be released in the next few months. Once `spark-3.5.10/` replaces `spark-3.5.9/` in https://dist.apache.org/repos/dist/release/spark, this suite silently stops testing Apache Spark 3.5 on Java 17 and still passes, because `(\d.\d.\d)` matches a single digit per component and doesn't match `spark-3.5.10/`. In addition, the string comparison with `SPARK_VERSION` is wrong for multi-digit components. For example, `"3.5.9" < "3.5.10-SNAPSHOT"` is `false` on `branch-3.5`. This was pointed out in https://github.com/apache/spark/pull/60049#discussion_r4451562553. ### Does this PR introduce _any_ user-facing change? No. This is a test-only change. ### How was this patch tested? Pass the CIs. In addition, I ran this suite manually on Java 17 with two release listings. This suite skips all old Spark versions on Java 21 and later. - The real listing, which has `spark-3.5.9/`, `spark-4.0.5/`, `spark-4.1.4/`, `spark-4.2.1/`, and `spark-4.3.0/`. - A local copy of it in which `spark-3.5.10/` replaces `spark-3.5.9/`. To skip the download, a copy of Apache Spark 3.5.9 is used as `spark-3.5.10` in `spark.test.cache-dir`. **Java 17 with `3.5.10` instead of `3.5.9`** ``` $ mkdir mirror $ curl -sL https://dist.apache.org/repos/dist/release/spark | sed 's/spark-3\.5\.9/spark-3.5.10/g' > mirror/spark $ cp -R $CACHE/spark-3.5.9 $CACHE/spark-3.5.10 $ SPARK_RELEASE_MIRROR=file://$PWD/mirror build/sbt -Phive -Dspark.test.cache-dir=$CACHE 'hive/testOnly *HiveExternalCatalogVersionsSuite' ... [info] HiveExternalCatalogVersionsSuite: [info] - backward compatibility (14 seconds, 899 milliseconds) ... [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` | Java | Release listing | Old Spark versions that ran `spark-submit` | `backward compatibility` | | :-: | :-: | :-: | :-: | | 17.0.21 | Real | 3.5.9, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (16 seconds) | | 17.0.21 | `3.5.10` instead of `3.5.9` | 3.5.10, 4.0.5, 4.1.4, 4.2.1, 4.3.0 | Passed (14 seconds) | For `3.5.10`, the copied Apache Spark 3.5.9 ran `spark-submit`, and the current build checked the tables of all five versions (`data_source_tbl_0` to `data_source_tbl_4`). Without this PR, the second row runs only 4.0.5, 4.1.4, 4.2.1, and 4.3.0, and the current build checks four versions. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
