> * the change is small: less then *100* source (only sources are counted - > not tests or other files) lines of code changed (90th percentile of the PRs > by external contributors) > * the change is localized. Localized means a few providers, docs or the > Helm chart. We estimate the CI minutes of the selected jobs from recent > runs, and the threshold is about 25% of a full run. This also means that > "task-sdk" PRs are **never** auto-approved - becuase task-sdk changes > trigger "full tests" > > Note that this is a bit temporary, provided that "Your PR. Your CI" will > work as expected. If we implement it,
These three bits added up means that right now a number of PRs will now be stuck behind an approval gate that maintainers are not expecting to be looking at. At the very least, this needs calling out in a separate email, not part they through a discuss thread, only that the expectation on Committers (to approve workflows) and on PR authors should have been called out in a separate email message > On 23 Oct 2026, at 11:03, Jarek Potiuk <[email protected]> wrote: > > Hello here, > > We are working with Shahar on finalising the "runs-on" setup and using > self-hosted runners (initially for canary runs only - but then later we > will expand it). > One follow-up to the self-hosted runners work before #74889 is merged. It's > about approving CI runs of PRs from forks. > > While reviewing the PR with Shahar, we determined that runner selection in > our workflows is a routing preference, not a security feature. A PR from a > fork runs its own workflow files, so it could ask for a RunsOn runner > directly. The real protection is on the AWS / RunsOn side. The other piece > is GitHub's "approve workflow runs" gate, which today applies only to > first-time contributors. > > We want to make that gate stricter ("Require approval for all external > contributors") BUT without making every contributor wait for a maintainer > and without flooding us with even more workflows from contributors. > > So we prepared an "Auto approve workflow runs" workflow (PR coming > shortly). Every 30 minutes it looks at the runs waiting for approval and > approves them when: > > * the PR does not touch .github/, dev/breeze/ or scripts/ci/, which define > the workflows and pick the runners (This is the security gate). > * we never auto-approve PRs from external contributos for other target than > `main` (i.e. PRs to v3-4-test need to be manually approved by maintainer) > * selective checks say it needs neither the full test set nor a dependency > upgrade > * the change is small: less then *100* source (only sources are counted - > not tests or other files) lines of code changed (90th percentile of the PRs > by external contributors) > * the change is localized. Localized means a few providers, docs or the > Helm chart. We estimate the CI minutes of the selected jobs from recent > runs, and the threshold is about 25% of a full run. This also means that > "task-sdk" PRs are **never** auto-approved - becuase task-sdk changes > trigger "full tests" > > Note that this is a bit temporary, provided that "Your PR. Your CI" will > work as expected. If we implement it, we won't really care if the PR is > small or not—we can approve the workflows anyway because (assuming CI is > not modified, which we will check), the "ASF" workflow will only run one or > two jobs; all other jobs will run using the contributors' CI. "Your PR. > Your CI" will work as expected. In case we implement it - we do not really > care if PR is small or not - we can approve the workflows anyway, because > (providing CI is not modified which we will check) - the "ASF" workflow > will **only** run one or two jobs - all the other jobs will be run using CI > of the contributors. > > Everything else gets a "*need workflow approval*" label, and it will need a > maintainer to approve it after looking at the diff (as it does today). And > we update contributor docs that contributor needs to personally ask for > their workflows to be approved in slack #new-contributors if they want it > to run. > > The scheduled workflow will only run code from main, never check out the > PR's code, and make a new decision for each push. Its summary shows the > expected jobs and the estimated CI time for every waiting PR. That should > also help when deciding whether to put "use self-hosted runners" on a PR. > > This is another small piece of our "gating" the flood of PRs. Of course > agents might automate this part, but I think it will be way easier to catch > those who abuse it. > > I checked in a scratch repo that approving with the workflow's GITHUB_TOKEN > starts the run right away, and workflow_run follow-ups still fire. So no > extra GitHub App or token is needed. > > Two notes: > > * Fork PRs still run on public runners. This changes who has to click > "approve", not where the runs go. > * Changing the approval setting needs ASF Infra - we have > > Unless there are objections, I'll treat this as part of the AIP-118 setup > and add a section to the AIP describing it. > > --------- > > Here is a simulation I've done using the last two weeks of PRs. We would > have auto-approved 42% of external contributors' runs with about 20% of > potential CI-minutes for jobs. > > At 310 job-minutes, the rules would have auto-approved 163 of the 349 > external fork PRs from 2026-10-09..23 (47%), about 11 a day. That's 30.5k > of the 141k job-minutes in the window; the other 110.5k wait for a > maintainer. Of the 114 merged PRs, 57 would have run without anyone > clicking approve. > > Decisions: > > Updated distribution (349 external PRs, 2026-10-09..23): > > ┌───────────────────────┬─────┬───────┐ > │ Decision │ PRs │ Share │ > ├───────────────────────┼─────┼───────┤ > │ Auto-approved │ 147 │ 42% │ > ├───────────────────────┼─────┼───────┤ > │ Over 310 job-minutes │ 122 │ 35% │ > ├───────────────────────┼─────┼───────┤ > │ Changes CI files │ 21 │ 6% │ > ├───────────────────────┼─────┼───────┤ > │ Needs full tests │ 21 │ 6% │ > ├───────────────────────┼─────┼───────┤ > │ Changes dependencies │ 16 │ 5% │ > ├───────────────────────┼─────┼───────┤ > │ Over 100 source lines │ 16 │ 5% │ > ├───────────────────────┼─────┼───────┤ > │ Not targeting main │ 6 │ 2% │ > └───────────────────────┴─────┴───────┘ > > Estimated job-minutes (approved / need approval): > > ┌─────────────┬──────────┬───────────────┐ > │ Job-minutes │ Approved │ Need approval │ > ├─────────────┼──────────┼───────────────┤ > │ 0–10 │ 7 │ 0 │ > ├─────────────┼──────────┼───────────────┤ > │ 10–100 │ 58 │ 3 │ > ├─────────────┼──────────┼───────────────┤ > │ 100–200 │ 7 │ 1 │ > ├─────────────┼──────────┼───────────────┤ > │ 200–300 │ 84 │ 4 │ > ├─────────────┼──────────┼───────────────┤ > │ 300–400 │ 7 │ 60 │ > ├─────────────┼──────────┼───────────────┤ > │ 400+ │ 0 │ 118 │ > └─────────────┴──────────┴───────────────┘ > > By area (approved / need approval; the last column shows how many were held > back by size alone): > > ┌───────────────────────┬──────────────────────────┬─────────────────────────┐ > │ Area │ Approved / need approval │ Held back by size > alone │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Core only │ 58 / 20 │ 4 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ UI │ 29 / 3 │ 2 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Providers only │ 24 / 70 │ 2 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Chart │ 10 / 1 │ 1 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Mixed areas │ 8 / 77 │ 5 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Language SDKs │ 3 / 2 │ 2 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Docs │ 6 / 0 │ 0 > │ > ├───────────────────────┼──────────────────────────┼─────────────────────────┤ > │ Task SDK, airflow-ctl │ 0 / 23 │ 0 (held back by cost) > │ > └───────────────────────┴──────────────────────────┴─────────────────────────┘ > > Auto-approval would cover about 20% of the estimated CI job-minutes for > external PRs, even though it covers 42% of the PRs. > > ┌──────────────────────────┬─────┬───────────────────────┬─────────┬──────────────────────┐ > │ │ PRs │ Estimated job-minutes │ Per day │ Share > of job-minutes │ > ├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤ > │ Auto-approved │ 147 │ 27,679 │ ~1,850 │ 19.6% > │ > ├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤ > │ Waiting for a maintainer │ 202 │ 113,368 │ ~7,560 │ 80.4% > │ > ├──────────────────────────┼─────┼───────────────────────┼─────────┼──────────────────────┤ > │ All external PRs │ 349 │ 141,047 │ ~9,400 │ 100% > │ > └──────────────────────────┴─────┴───────────────────────┴─────────┴──────────────────────┘ > > The cheap PRs (UI, docs, single core areas, small providers) run > independently. The remaining 80% of job-minutes belong to expensive runs > (Task SDK, full tests, large providers, multi-area changes), and those wait > for a maintainer's click. > > How this was calculated: > - One run per PR. We estimate one run per PR's latest commit, but every > push starts a new run. Actual totals will be higher by the average number > of pushes per PR, and this ratio holds only if cheap and expensive PRs are > pushed about equally often. > - External fork PRs only. Committers' PRs (527 of the 876 PRs in the two > weeks) never wait for approval and aren't included. > - Public runners. The numbers are for Tests (AMD) on GitHub-hosted runners. > > J. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
