andygrove opened a new pull request, #6656: URL: https://github.com/apache/datafusion-comet/pull/6656
## Which issue does this PR close? No issue. This came out of looking at why CI on #6652, which only moves test coverage into a Comet SQL fixture, ran the Rust tests. ## Rationale for this change `linux-test-rust` runs whenever `build_linux` matches, and `build_linux` matches all of `spark/**`. The job itself compiles only `native/` and the `common` module, whose classes the JNI tests load, and nothing under `native/` reads from `spark/`. So every pull request confined to the Spark module, main code or tests, spends about 20 runner-minutes a push on a job it cannot affect, and the same again in the merge queue. On the first push of #6652 that was 20 of 168 runner-minutes. The SQL test migration PRs under #6615 all have this shape. This PR leaves the other jobs a test-only change can't affect alone, such as the TPC checks or the suite groups it doesn't touch. A main-code change under `spark/` can affect those, so narrowing them needs more than a path filter. ## What changes are included in this PR? - `dev/ci/compute-changes.py`: a new `rust_test` output with its own path filter. It covers `native/`, `common/`, the root `pom.xml`, the Maven wrapper, `rust-toolchain.toml`, `dev/ci/`, the two workflow files and the three actions the job uses, with the same markdown and bench excludes as `build_linux`. Its policy is `pr` and `queue`, like `build_linux_full`. - `.github/workflows/ci.yml` exports the output and passes it to `pr_build_linux.yml` as a new `rust-tests` input, which `linux-test-rust` now checks in its `if:`. The input defaults to `true`, so a caller that omits it runs the job rather than dropping it. - `dev/ci/check-ci-config.py`: routing cases for native, common, the toolchain, the root pom, the Rust action, Spark main code and a Spark SQL fixture, with `rust_test` added to the PR tier and the Maven wrapper cases. There is also a new check that every routing output is exported by the `Detect changes` job and read in `ci.yml`. Without it, a missing export line reads as false and the job it gates stops running everywhere, with nothing failing. - `docs/source/contributor-guide/ci.md` and `.github/workflows/README.md` describe the new filter. Push to main and the nightly are unchanged, since the job was already skipped there. A workflow dispatch still runs everything. One trade-off: `RUST_VERSION` is `stable`, so today a Rust release with new lints or deprecations turns every PR red at once, as 1.99 did before #6507. After this change only PRs that touch `native/` or `common/` see it. `Build Native Library` still compiles on every PR, so a toolchain that can't build the code is still caught everywhere. ## How are these changes tested? - `python3 dev/ci/check-ci-config.py` passes. I also broke the wiring nine ways in a scratch copy: dropping the export, misspelling the read, dropping the `rust-tests:` line, forwarding the wrong step output, removing `common/**`, `pom.xml` or `.mvn/**` from the filter, adding `spark/**`, and putting the output on push. Each one fails the check with a message that names the cause. - `compute-changes.py` on #6652's two files returns `rust_test=false`, with `build_linux` and `build_linux_full` still true. On a native file it returns `true` for `pull_request`, `merge_group` and `workflow_dispatch`, and `false` for `push` and `schedule`. - `actionlint`, prettier and RAT pass. This PR's own run includes the Rust job, because it changes the workflow files, so it exercises the `true` path. The skip shows up on the first `spark/`-only PR after this merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
