(and sorry for [Recipient Name]). By a popular request, I use Gemini to make my mails shorter (the original was 2x as long) - and sometimes it does those things :D
On Fri, Oct 16, 2026 at 1:32 PM Jarek Potiuk <[email protected]> wrote: > Hi [Recipient Name], > > Regarding the question about runners: that is actually not how GitHub > Actions works by default. Pull request workflows always execute using the > target repository's runners. You have to custom-engineer your workflows to > run checks on a fork's runners. > > In Airflow, we are working on a setup where pull_request workflows > immediately skip all checks on the main repo, and the checks instead run > via push triggers on the fork. > > While the basic idea sounds straightforward, implementation is complex. > Because the workflow runs in the fork, its status won't automatically sync > with the Apache PR. You have to reconstruct that link manually—using > workflow_run or similar mechanisms—to post custom statuses and links back > to the PR in the target repository. > > For reference, this is not for the faint of heart, but you can look at our > design in Airflow: > > - Airflow PR: https://github.com/apache/airflow/pull/73378 (contains > links to related PRs and AIP-120 describing the approach) > > I hope to have this running in a pilot mode next week. Once validated, I > will share our learnings and potentially propose shared actions in > infrastructure-actions so other projects can set this up more easily. > > I also plan to share our analysis with GitHub - they are on a spree to fix > things for maintainers. A key side effect of this setup is that free > accounts are limited to 20 parallel runners. For large PRs with many matrix > jobs, contributors will iterate more slowly (or queue longer). However, > for small PRs with selective testing, it should be just as fast—or even > faster, since they won't queue behind other Apache projects. > > Fingers crossed we will not find some unexpected issues and we can pull it > off and share with others. > > Best regards, > Jarek > > > On Fri, Oct 16, 2026 at 12:08 PM Arnout Engelen <[email protected]> > wrote: > >> >> From: Kevin Liu <[email protected]> >>> Date: Thu, Oct 15, 2026 at 6:07 PM >>> Subject: GitHub Actions: one user's PR burst on grails-core is >>> saturating ubuntu-24.04 for the whole org >>> To: <[email protected]> >>> >>> >>> Hi Infra team, >>> >>> Linux runners in the apache org have been saturated for the past two >>> days, and PR CI across many projects is waiting hours for a runner. >>> >>> What's happening right now: A single user opened a batch of about 17 >>> PRs on a single project, apache/grails-core, around 13:07 UTC today. >>> Each PR's CI run is using 25–47 jobs at once. Together they hold 569 of >>> the 576 ubuntu-24.04 runners in use across the org. >>> >> I haven't checked myself, but I heard that when you open a PR from your >> own fork instead of from a branch in the apache org, it gets the runners >> from your own (free) budget instead of from the ASF's runner pool. If I'm >> seeing it correctly, the 17 PRs in question are indeed coming from branches >> under the apache org. Can anyone confirm my understanding? If that's indeed >> the case, should we be recommending committers to create PRs from their own >> forks rather than from branches under the apache org more actively? >> >> >> Cheers, >> >> Arnout >> >> >>> That's above the 20-job limit in the GitHub Actions policy >>> <https://infra.apache.org/github-actions-policy.html>. >>> >>> Everyone else on ubuntu-24.04 is stuck behind it, including iceberg, >>> zeppelin, datafusion-comet, ozone and iceberg-rust. Snapshot at 16:56 UTC >>> across the 345 apache repos pushed to in the last 24 hours: >>> Label Running Queued Median wait of queued jobs >>> ubuntu-latest 86 1,617 41 min >>> ubuntu-24.04 576 1,095 129 min >>> ubuntu-26.04 105 439 67 min >>> >>> Why we care: iceberg-rust's merge queue was timing out on runner waits ( >>> apache/iceberg-rust#3442 >>> <https://github.com/apache/iceberg-rust/issues/3442>). Yesterday we >>> moved off ubuntu-latest to ubuntu-24.04, which cut our median CI time >>> from 232 minutes to 9 minutes. Since about 15:00 UTC today we're back to >>> queuing for over an hour. >>> >>> Asks: >>> >>> 1. >>> >>> Can you reach out to grails-core about capping matrix concurrency >>> (strategy.max-parallel: >>> 15 or less)? >>> 2. >>> >>> Is the 20-job limit enforced or monitored anywhere? A burst like >>> this from one user shouldn't be able to take over the shared pool. >>> 3. >>> >>> Separately, ubuntu-latest has had a backlog of 1,600–2,000 jobs for >>> two days, even while other labels were idle. Is there something specific >>> to >>> that label you can see on your side? >>> >>> Happy to share the scripts and raw data. >>> >>> Best, >>> >>> Kevin >>> >> >> >> -- >> Arnout Engelen >> ASF Security Response >> Apache Pekko PMC member, ASF Member >> NixOS Committer >> Independent Open Source consultant >> >
