> From: Kevin Liu <[email protected]> > Date: Thu, Oct 15, 2026 at 6:07 PM > Subject: GitHub Actions: one user's PR burst on grails-core is saturating > ubuntu-24.04 for the whole org > To: <[email protected]> > > > Hi Infra team, > > Linux runners in the apache org have been saturated for the past two > days, and PR CI across many projects is waiting hours for a runner. > > What's happening right now: A single user opened a batch of about 17 PRs > on a single project, apache/grails-core, around 13:07 UTC today. Each PR's > CI run is using 25–47 jobs at once. Together they hold 569 of the 576 > ubuntu-24.04 runners in use across the org. > I haven't checked myself, but I heard that when you open a PR from your own fork instead of from a branch in the apache org, it gets the runners from your own (free) budget instead of from the ASF's runner pool. If I'm seeing it correctly, the 17 PRs in question are indeed coming from branches under the apache org. Can anyone confirm my understanding? If that's indeed the case, should we be recommending committers to create PRs from their own forks rather than from branches under the apache org more actively?
Cheers, Arnout > That's above the 20-job limit in the GitHub Actions policy > <https://infra.apache.org/github-actions-policy.html>. > > Everyone else on ubuntu-24.04 is stuck behind it, including iceberg, > zeppelin, datafusion-comet, ozone and iceberg-rust. Snapshot at 16:56 UTC > across the 345 apache repos pushed to in the last 24 hours: > Label Running Queued Median wait of queued jobs > ubuntu-latest 86 1,617 41 min > ubuntu-24.04 576 1,095 129 min > ubuntu-26.04 105 439 67 min > > Why we care: iceberg-rust's merge queue was timing out on runner waits ( > apache/iceberg-rust#3442 > <https://github.com/apache/iceberg-rust/issues/3442>). Yesterday we moved > off ubuntu-latest to ubuntu-24.04, which cut our median CI time from 232 > minutes to 9 minutes. Since about 15:00 UTC today we're back to queuing for > over an hour. > > Asks: > > 1. > > Can you reach out to grails-core about capping matrix concurrency > (strategy.max-parallel: > 15 or less)? > 2. > > Is the 20-job limit enforced or monitored anywhere? A burst like this > from one user shouldn't be able to take over the shared pool. > 3. > > Separately, ubuntu-latest has had a backlog of 1,600–2,000 jobs for > two days, even while other labels were idle. Is there something specific to > that label you can see on your side? > > Happy to share the scripts and raw data. > > Best, > > Kevin > -- Arnout Engelen ASF Security Response Apache Pekko PMC member, ASF Member NixOS Committer Independent Open Source consultant
