yihua opened a new pull request, #19082:
URL: https://github.com/apache/hudi/pull/19082
### Change Logs
Reduce CI cost by tiering the Spark datasource test matrix: run a curated
**core** subset on every Spark version, and the **full** unit/functional/scala
suites only on the latest 3.x (spark3.5 / Scala 2.12 / Java 11) and latest 4.x
(spark4.2 / Scala 2.13 / Java 17). The full suites for non-latest versions are
mostly Spark-version-independent and are the bulk of CI time.
Mechanism:
- New `core-tests` Maven profile: surefire `groups=core` plus scalatest
`tagsToInclude=org.apache.hudi.functional.SparkSQLCoreFlow`, injected only in
this profile (an empty `tagsToInclude` would emit `-n ""` and run zero scala
tests).
- The core set is excluded from the normal tasks so nothing runs twice on a
version: `core` added to `excludedGroups` of
`unit-tests`/`functional-tests`/`-b`/`-c`, and the scalatest exclude is driven
by the `hoodie.scalatest.tagsToExclude` property (default `SparkSQLCoreFlow`).
- `CoreFlow` scalatest `Tag` object (name matches the existing
`@SparkSQLCoreFlow` `@TagAnnotation`) for per-test `taggedAs(CoreFlow)`.
- Tagged core set: `@Tag("core")` on representative
`TestCOWDataSource`/`TestMORDataSource` methods (read/write round-trip,
vectorized reader, snapshot/read-optimized/incremental reads, schema evolution)
and the parquet/orc/InternalRow writers; `taggedAs(CoreFlow)` on basic
INSERT/UPDATE/DELETE/MERGE INTO/CREATE TABLE blocks.
- The previously dead-wired `TestSparkSqlCoreFlow` anchor (tagged
`@SparkSQLCoreFlow`, excluded everywhere, included nowhere) is now run by the
core job, trimmed to a representative table-type/metadata/keygen/index spread
so it fits the per-version budget.
- `bot.yml`: full-suite datasource jobs reduced to a single version each;
two new jobs `test-spark-core-tests` (spark3.3/3.4/3.5) and
`test-spark-java17-core-tests` (spark3.5/4.0/4.1/4.2) build once and run
`-Pcore-tests` plus the quickstart on every version. Flink (already tiered),
bundle-validation, docker, and build-only jobs are unchanged.
Estimated impact (from measured per-job timings on a recent master run):
~2,119 -> ~1,142 runner-minutes per push (~46%), and Spark datasource checks
drop from 30 to ~14.
### Impact
CI/test-execution only; no production code or public API change. Non-latest
Spark versions trade the full per-version suite for a curated core subset; full
coverage for all versions can still be obtained via a scheduled full-matrix run.
### Risk level (write none, low medium or high below)
low
This is a CI/test-tiering change. Opened as draft to validate the new matrix
in CI.
### Documentation Update
None.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact described above
- [ ] CI passing on the new matrix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
