Hello here,
We are working with Shahar on finalising the "runs-on" setup and using
self-hosted runners (initially for canary runs only - but then later we
will expand it).
One follow-up to the self-hosted runners work before #74889 is merged. It's
about approving CI runs of PRs from forks.
While reviewing the PR with Shahar, we determined that runner selection in
our workflows is a routing preference, not a security feature. A PR from a
fork runs its own workflow files, so it could ask for a RunsOn runner
directly. The real protection is on the AWS / RunsOn side. The other piece
is GitHub's "approve workflow runs" gate, which today applies only to
first-time contributors.
We want to make that gate stricter ("Require approval for all external
contributors") BUT without making every contributor wait for a maintainer
and without flooding us with even more workflows from contributors.
So we prepared an "Auto approve workflow runs" workflow (PR coming
shortly). Every 30 minutes it looks at the runs waiting for approval and
approves them when:
* the PR does not touch .github/, dev/breeze/ or scripts/ci/, which define
the workflows and pick the runners (This is the security gate).
* we never auto-approve PRs from external contributos for other target than
`main` (i.e. PRs to v3-4-test need to be manually approved by maintainer)
* selective checks say it needs neither the full test set nor a dependency
upgrade
* the change is small: less then *100* source (only sources are counted -
not tests or other files) lines of code changed (90th percentile of the PRs
by external contributors)
* the change is localized. Localized means a few providers, docs or the
Helm chart. We estimate the CI minutes of the selected jobs from recent
runs, and the threshold is about 25% of a full run. This also means that
"task-sdk" PRs are **never** auto-approved - becuase task-sdk changes
trigger "full tests"
Note that this is a bit temporary, provided that "Your PR. Your CI" will
work as expected. If we implement it, we won't really care if the PR is
small or not—we can approve the workflows anyway because (assuming CI is
not modified, which we will check), the "ASF" workflow will only run one or
two jobs; all other jobs will run using the contributors' CI. "Your PR.
Your CI" will work as expected. In case we implement it - we do not really
care if PR is small or not - we can approve the workflows anyway, because
(providing CI is not modified which we will check) - the "ASF" workflow
will **only** run one or two jobs - all the other jobs will be run using CI
of the contributors.
Everything else gets a "*need workflow approval*" label, and it will need a
maintainer to approve it after looking at the diff (as it does today). And
we update contributor docs that contributor needs to personally ask for
their workflows to be approved in slack #new-contributors if they want it
to run.
The scheduled workflow will only run code from main, never check out the
PR's code, and make a new decision for each push. Its summary shows the
expected jobs and the estimated CI time for every waiting PR. That should
also help when deciding whether to put "use self-hosted runners" on a PR.
This is another small piece of our "gating" the flood of PRs. Of course
agents might automate this part, but I think it will be way easier to catch
those who abuse it.
I checked in a scratch repo that approving with the workflow's GITHUB_TOKEN
starts the run right away, and workflow_run follow-ups still fire. So no
extra GitHub App or token is needed.
Two notes:
* Fork PRs still run on public runners. This changes who has to click
"approve", not where the runs go.
* Changing the approval setting needs ASF Infra - we have
Unless there are objections, I'll treat this as part of the AIP-118 setup
and add a section to the AIP describing it.
---------
Here is a simulation I've done using the last two weeks of PRs. We would
have auto-approved 42% of external contributors' runs with about 20% of
potential CI-minutes for jobs.
At 310 job-minutes, the rules would have auto-approved 163 of the 349
external fork PRs from 2026-10-09..23 (47%), about 11 a day. That's 30.5k
of the 141k job-minutes in the window; the other 110.5k wait for a
maintainer. Of the 114 merged PRs, 57 would have run without anyone
clicking approve.
Decisions:
Updated distribution (349 external PRs, 2026-10-09..23):
┌───────────────────────┬─────┬───────┐
│ Decision │ PRs │ Share │
├───────────────────────┼─────┼───────┤
│ Auto-approved │ 147 │ 42% │
├───────────────────────┼─────┼───────┤
│ Over 310 job-minutes │ 122 │ 35% │
├───────────────────────┼─────┼───────┤
│ Changes CI files │ 21 │ 6% │
├───────────────────────┼─────┼───────┤
│ Needs full tests │ 21 │ 6% │
├───────────────────────┼─────┼───────┤
│ Changes dependencies │ 16 │ 5% │
├───────────────────────┼─────┼───────┤
│ Over 100 source lines │ 16 │ 5% │
├───────────────────────┼─────┼───────┤
│ Not targeting main │ 6 │ 2% │
└───────────────────────┴─────┴───────┘
Estimated job-minutes (approved / need approval):
┌─────────────┬──────────┬───────────────┐
│ Job-minutes │ Approved │ Need approval │
├─────────────┼──────────┼───────────────┤
│ 0–10 │ 7 │ 0 │
├─────────────┼──────────┼───────────────┤
│ 10–100 │ 58 │ 3 │
├─────────────┼──────────┼───────────────┤
│ 100–200 │ 7 │ 1 │
├─────────────┼──────────┼───────────────┤
│ 200–300 │ 84 │ 4 │
├─────────────┼──────────┼───────────────┤
│ 300–400 │ 7 │ 60 │
├─────────────┼──────────┼───────────────┤
│ 400+ │ 0 │ 118 │
└─────────────┴──────────┴───────────────┘
By area (approved / need approval; the last column shows how many were held
back by size alone):
┌───────────────────────┬──────────────────────────┬─────────────────────────┐
│ Area │ Approved / need approval │ Held back by size
alone │
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Core only │ 58 / 20 │ 4
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ UI │ 29 / 3 │ 2
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Providers only │ 24 / 70 │ 2
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Chart │ 10 / 1 │ 1
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Mixed areas │ 8 / 77 │ 5
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Language SDKs │ 3 / 2 │ 2
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Docs │ 6 / 0 │ 0
│
├───────────────────────┼──────────────────────────┼─────────────────────────┤
│ Task SDK, airflow-ctl │ 0 / 23 │ 0 (held back by cost)
│
└───────────────────────┴──────────────────────────┴─────────────────────────┘
Auto-approval would cover about 20% of the estimated CI job-minutes for
external PRs, even though it covers 42% of the PRs.
┌──────────────────────────┬─────┬───────────────────────┬─────────┬──────────────────────┐
│ │ PRs │ Estimated job-minutes │ Per day │ Share
of job-minutes │
├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤
│ Auto-approved │ 147 │ 27,679 │ ~1,850 │ 19.6%
│
├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤
│ Waiting for a maintainer │ 202 │ 113,368 │ ~7,560 │ 80.4%
│
├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤
│ All external PRs │ 349 │ 141,047 │ ~9,400 │ 100%
│
└──────────────────────────┴─────┴───────────────────────┴─────────┴──────────────────────┘
The cheap PRs (UI, docs, single core areas, small providers) run
independently. The remaining 80% of job-minutes belong to expensive runs
(Task SDK, full tests, large providers, multi-area changes), and those wait
for a maintainer's click.
How this was calculated:
- One run per PR. We estimate one run per PR's latest commit, but every
push starts a new run. Actual totals will be higher by the average number
of pushes per PR, and this ratio holds only if cheap and expensive PRs are
pushed about equally often.
- External fork PRs only. Committers' PRs (527 of the 876 PRs in the two
weeks) never wait for approval and aren't included.
- Public runners. The numbers are for Tests (AMD) on GitHub-hosted runners.
J.
