2010YOUY01 opened a new pull request, #23414:
URL: https://github.com/apache/datafusion/pull/23414
### Summary
This PR introduces an easy way to reproduce CI checks locally.
To reproduce `.github/workflows/rust.yml`:
```sh
cargo xtask ci workflow rust
```
Example output:
```sh
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci
workflow rust
ci workflow `rust` total execution time: 565.131385s
ci job `linux-rustdoc`: 210.992471s
ci job `linux-test-doc`: 117.395768s
ci job `linux-test`: 85.890313s
ci job `linux-test-example`: 39.367084s
ci job `linux-test-datafusion-cli`: 37.235765s
ci job `clippy`: 24.750692s
ci job `msrv`: 16.421963s
ci job `macos-aarch64`: 12.399443s
ci job `config-docs-check`: 6.801263s
ci job `check-fmt`: 3.139787s
ci job `linux-cargo-check-datafusion`: 2.715153s
ci job `cargo-toml-formatting-checks`: 1.94445s
ci job `examples-docs-check`: 1.518475s
ci job `linux-cargo-check-datafusion-functions`: 1.121708s
ci job `sqllogictest-substrait`: 991.08ms
ci job `linux-build-lib`: 600.417ms
ci job `linux-datafusion-proto-features`: 596.001ms
ci job `linux-datafusion-substrait-features`: 528.707ms
ci job `vendor`: 346.94ms
ci job `linux-datafusion-common-features`: 208.197ms
ci job `verify-clean`: 83.283ms
ci job `verify-clean`: 82.425ms
```
To reproduce single job or step similarly,
```sh
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci
job check-fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`
ci job `check-fmt` total execution time: 2.817591s
ci step `fmt`: 2.817591s
yongting@Yongtings-MacBook-Pro-2 ~/C/datafusion (ci-xtask)> cargo xtask ci
step fmt
+ /Users/yongting/Code/datafusion/ci/scripts/rust_fmt.sh
[rust_fmt.sh] `cargo fmt --all -- --check`
ci step `fmt` completed in 2.793302s
```
### Background
The existing GitHub CI follows a 3-layer structure:
- step: an atomic check, for example `cargo fmt`
- job: one or more steps, for example `check-fmt`
- workflow: a list of jobs, for example `.github/workflows/rust.yml`
For example:
```text
workflow: rust
job: check-fmt
step: fmt
command: cargo fmt --all -- --check
job: clippy
step: clippy
command: cargo clippy ...
job: linux-test
step: rust-test
command: cargo test ...
step: verify-clean
command: git diff --exit-code
```
This PR mirrors that structure in `xtask`, so the same CI units can be used
by both GitHub Actions and local runs.
### Motivation
Making CI easy to reproduce locally improves developer experience.
- If a specific job fails on GitHub, we can reproduce it locally with `cargo
xtask ci job <job>`.
- After a small change, we can run a subset of the full CI locally. This
shortens the development cycle while still giving good confidence that the full
CI will pass.
Possible future extensions:
```sh
cargo xtask ci workflow changed
```
Detect changed Rust files, then only run unit tests and clippy on crates
affected by the change, plus relevant integration tests.
```sh
cargo xtask ci workflow doc
```
For documentation-only changes, only run formatting, typo checks, docs
checks, and other lightweight checks.
### Implementation
The implementation adds reusable units for atomic checks, such as `cargo
fmt`. In GitHub Actions terminology, these are "steps".
The key idea is:
- GitHub CI orchestrates these atomic checks remotely.
- `cargo xtask ci ...` orchestrates the same checks locally.
There are two viable ways to organize these atomic checks:
- `cargo xtask`, for example `cargo xtask ci step fmt`
- shell scripts, for example `./check_fmt.sh`
I previously tried the shell script approach for `dev.yml`:
https://github.com/apache/datafusion/blob/main/dev/rust_lint.sh
For the current CI complexity, I found `cargo xtask` easier to keep
organized. I suggest we continue with this approach and later migrate the
existing scripts to use `xtask` as well.
### Scope of this PR
This PR:
- Introduces the `xtask` framework.
- Uses the `ci` domain for CI-related checks, for example `cargo xtask ci
...`.
- Migrates the steps in `rust.yml` to `xtask` runners.
- Moves commands 1:1 from GitHub Actions to `xtask`, without changing the
commands, except for a few small differences noted below.
- Supports local orchestration with `cargo xtask ci workflow rust`.
Notes:
- This PR only achieves CI check command equivalence. Setup steps and tool
installation are not fully reproduced yet, and are expected to be done manually
for now.
- Three jobs are skipped in the local run because they need slightly
different handling on macOS. I would like to add them in follow-up PRs to keep
this PR easier to review.
### Testing
Individual 'step's will be tested by CI.
For workflow/jobs, I have tested locally on MacOS. I think we can let CI to
test some fast workflows in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
