andygrove opened a new pull request, #7483:
URL: https://github.com/apache/datafusion-comet/pull/7483
## Which issue does this PR close?
None yet.
## Rationale for this change
The `linux-test` and macOS test matrices name all 180 Comet suites in two
YAML lists that have to
be kept in sync by hand, and a new suite has to be added to both. Only
GitHub Actions can read the
lists, so running a CI group locally or on another CI system means copying
them and keeping the
copy current.
Moving group membership into the suites, as ScalaTest tags, and the Maven
invocation into a script
gives one definition of each group that any runner can use.
## What changes are included in this PR?
- Four tag annotations in `org.apache.comet.tags`: `ScansGroup`,
`ShuffleGroup`, `ExecGroup` and
`ExpressionsGroup`. Each concrete suite carries the tag for the group it
was listed in. The 180
suites map to the same groups as before, checked mechanically against the
old lists.
- `dev/ci/run-comet-tests.sh <group> [maven args]` runs
`mvnw test -DtagsToInclude=org.apache.comet.tags.<Group>`. The `java-test`
action calls it with
`--goal install -Prelease`, so CI runs the same goal and profile as before.
- The `suite` matrix in `pr_build_linux.yml` and `pr_build_macos.yml` is now
the four group names.
- `dev/ci/check-suites.py` checks that every concrete suite under
`spark/src/test` has exactly one
group tag, including each per-Spark-version copy, and that the suites it
ignores have none.
- The development guide and the PR review skill describe the tags in place
of the lists.
One behavior difference: name filtering (`-DwildcardSuites`) never loads the
other groups' suites,
while tag filtering instantiates every suite and runs only the tagged tests.
ScalaTest skips
`beforeAll` for a suite with no selected tests, so no SparkSession starts. A
local run that selects
no tests across all 188 discovered suites takes 19 seconds.
## How are these changes tested?
- `check-suites.py` and `check-ci-config.py` pass.
- `test-compile` passes with the Spark 3.4, 3.5, 4.0, 4.1 and 4.2 profiles.
- Locally, `-DtagsToInclude=org.apache.comet.tags.ExpressionsGroup` with a
wildcard naming one
`expressions` suite and one `exec` suite runs the first suite's 22 tests
and none of the second's.
- This PR's CI runs the four groups through the new script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
