kumarUjjawal opened a new pull request, #26760: URL: https://github.com/apache/datafusion/pull/26760
## Which issue does this PR close? Related to #25148. Inspired by apache/airflow#75619. ## Rationale for this change Merge queue runs continue using runners after their group is removed or replaced. The Rust workflow gives each merge-group SHA a separate concurrency group, so a replacement does not cancel the old run. Jobs still waiting for a runner can start long after the queue has stopped using their results. A read-only analysis of runs created on October 19–25, 2026 (UTC) found: | Measurement | Result | | --- | ---: | | Merge-group workflow runs inspected | 1,170 | | Distinct merge-group commits / PRs | 195 / 154 | | Removed or superseded groups with subsequent runner use | 36 | | Workflows / jobs with subsequent runner use | 91 / 473 | | Runner time after removal or replacement | 20.3 hours | | Jobs that started after removal or replacement | 312, using 12.7 runner-hours | For the Rust and extended-test workflows, this was 19.0 of 369.4 runner-hours, or 5.2% of their sampled runner time. A cancellation taking effect one minute after obsolescence would have avoided 17.9 runner-hours across the affected workflows; at five minutes, 11.9 hours; at fifteen minutes, 9.4 hours. Examples: - [#26562](https://github.com/apache/datafusion/pull/26562): after manual queue removal, 34 jobs used another 121.6 runner-minutes in parallel. They finished within about 14 minutes. [Rust run](https://github.com/apache/datafusion/actions/runs/40069843292). - [#26373](https://github.com/apache/datafusion/pull/26373): the group was removed for failed checks at 21:21 UTC on October 22, but its MSRV job did not start until 23:20 UTC. [Rust run](https://github.com/apache/datafusion/actions/runs/39855621556). The cutoff was an exact-SHA queue-removal event or the creation of a replacement group for the same PR. For each job, the analysis summed time after the later of its start and the cutoff plus the assumed cancellation delay. It excluded skipped jobs, deduplicated successful jobs copied into rerun records, and excluded seven rerun attempts started after obsolescence that could have been deliberate debugging. All Rust and extended-test jobs were fetched for the denominator; other workflows were inspected when their run metadata showed activity after the cutoff. These are measured historical runner times and estimated avoidable portions, not observed savings from deploying this workflow or billing estimates. The analysis did not measure queue-ref deletion latency, cancellation cleanup, scheduler delays, or this workflow's overhead. Recent runner changes also make this sample unsuitable as an annual forecast. ## What changes are included in this PR? - Run cleanup when a new merge group is created and every 15 minutes, with a manual dry-run option. - Only target unfinished `merge_group` runs in `apache/datafusion` whose exact queue ref is missing or points to another commit. Recheck the ref and run before requesting cancellation; preserve active groups, other event types, and cleanup itself. - Execute the script from the default branch with `contents: read` and `actions: write`. API errors preserve the affected run and do not prevent other candidates from being checked. Recheck HTTP 409 responses so failed cancellations are not silently treated as completed runs. - Add regression tests to both Dev CI and the local lint suite, and document operation and deliberate reruns. ## What is the testing strategy for this PR? - 26 cleanup tests cover active groups, replaced/missing refs, all unfinished statuses, pagination, dry runs, API failures, and completion/rerun races. The HTTP 409 regression fails against the earlier implementation. Together with the existing fork-CI tests, 50 Node tests pass. - `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and the complete `./dev/rust_lint.sh` suite pass, including generated docs, links, dependency checks, Rust docs, and the HTML documentation build. The security audit reports only its two existing allowed warnings. - actionlint 1.7.12 passes for both changed workflows. - A live read-only dry run completed successfully with no currently eligible runs. No workflow was cancelled during validation. Production cancellation latency remains untested. ## Are there any user-facing changes? Obsolete merge-queue CI runs will be cancelled automatically. Contributors can run CI manually on a branch when investigating an obsolete commit. No query behavior or public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
