kumarUjjawal opened a new pull request, #25365:
URL: https://github.com/apache/datafusion/pull/25365
## Which issue does this PR close?
- Part of #25148. This PR does not close the issue.
## Rationale for this change
Every commit reaches upstream `main` through the merge queue. The queue runs
the Rust workflow at that commit. The push to `main` then runs the same 26 jobs
at the same SHA.
Only seven of those jobs give new output after the merge:
- `linux-build-lib`, `linux-cargo-check-datafusion-functions`, `linux-test`,
`linux-test-example`, `clippy`, and `macos-aarch64` save the caches that later
PRs restore. The workflow saves caches only on pushes to `main`.
- `linux-test` publishes coverage.
- `linux-datafusion-ffi-features` is not a required check in `.asf.yaml`, so
a PR can merge when it fails. The post-merge run shows that failure on `main`.
From September 9 to September 16, 2026, the Rust workflow ran 116 times for
pushes to `main`. Each of those commits had a successful merge-queue run at the
same SHA before the push. The ten most recent pushes used a mean of 71.7
runner-minutes each. The 19 skipped jobs account for 47.6 of those minutes:
19.5 on GitHub-hosted Linux runners and 28.0 on RunsOn. At that merge rate,
this change saves about 2,200 GitHub-hosted Linux runner-minutes and 3,100
RunsOn runner-minutes per week. The GitHub-hosted minutes count against the
shared ASF limit that the issue describes.
Tradeoff: a direct push to `main` that bypasses the merge queue also skips
the 19 jobs. A manual `workflow_dispatch` on `main` runs all 26 jobs.
`extended.yml` already uses the same approach for `main`.
Post-merge runs also catch intermittent failures. In the sample week, three
post-merge runs failed. Each commit had passed the same jobs in the queue. The
logs show environment errors: a test memory-pool limit, a ChromeDriver port
bind, and a MinIO image pull. Two of the three failed jobs are in the skipped
set.
## What changes are included in this PR?
`.github/workflows/rust.yml`:
- 19 jobs get `if: ${{ !(github.event_name == 'push' && github.ref ==
'refs/heads/main' && github.repository == 'apache/datafusion') }}`. The
condition has no status function, so jobs that depend on `linux-build-lib`
still require that job to succeed.
- The Cargo check artifact archive and upload steps in `linux-build-lib` get
the same condition. Both consumers of that artifact are in the skipped set.
- A header comment explains the behavior.
| Event | Rust jobs |
| ---------------------------------------- | --------: |
| Push to upstream `main` | 7 |
| PR or merge group | 26 |
| `workflow_dispatch` | 26 |
| Push to another branch, or a fork push | 26 |
`ci/scripts/check_asf_yaml_status_checks.py` gets a second check. It
requires that exactly the jobs in `POST_MERGE_SKIPPED_JOBS` carry the
condition, that the artifact steps carry it, and that every job that downloads
the `cargo-check` artifact is in the skipped set. A new job without the
condition passes, so new jobs run by default.
Other files:
- `ci/scripts/test_check_asf_yaml_status_checks.py`: unit tests for the new
check.
- `.github/workflows/dev.yml`: the `asf-yaml-check` job runs the unit tests.
- `docs/source/contributor-guide/testing.md`: one sentence on the new check.
- `.gitignore`: `__pycache__/`.
## What is the testing strategy for this PR?
`python3 ci/scripts/check_asf_yaml_status_checks.py` passes on this branch.
`python3 -m unittest discover --start-directory ci/scripts` runs 18 tests.
The tests load the real `rust.yml` and make sure that:
- Exactly the listed jobs carry the condition.
- The seven kept jobs have no condition.
- The condition contains no status function.
- The push trigger has a branch filter, so tags never trigger the workflow.
- The condition skips a job only for a push to `refs/heads/main` in
`apache/datafusion`. PRs, merge groups, `workflow_dispatch`, other branches,
and forks run the job.
Fixture tests make sure that the check rejects:
- A listed job without the condition.
- A kept job with the condition.
- A renamed job.
- A changed event, ref, or repository in the condition.
- An added `always()`.
- An artifact step without its condition.
- An artifact consumer outside the skipped set.
Hosted CI on this PR runs all 26 Rust jobs. The first push to `main` after
the merge is the check of the reduced path: seven Rust jobs run and 19 skip.
## Are there any user-facing changes?
No. This PR changes CI and the contributor guide only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
