I've created https://github.com/apache/grails-core/pull/16673 to help reduce the consumption of runners. It ships with a new policy that decides when to run CI. If people agree, I'd like to merge this so we can label some of these PRs as 'no ci' and they'll stop consuming so much time.
Separately, I've been looking at the self hosted runners, and it looks like pricing will be an issue. We will need infrastructure's approval to use the AWS money. I'll have a more detailed write up in the next few days with a proposal for how it should work. On Fri, Oct 16, 2026 at 9:18 AM James Daugherty <[email protected]> wrote: > @jamesfredley who do we need permission from to use the AWS subaccount to > host self hosted agents? > > I think it's time we switch - it would help us and others. > > On 2026-10-16 06:20 UTC Robert Thomson wrote: > > Hi, > > > > Please see this post in [email protected] mailing list regarding GitHub > > Actions usage. > > If this usage is something that is required, please consider looking for > > donations that can be used to run self-hosted runners. > > > > Thanks. > > Kind regards, > > -Bob Thomson. > > > > > > ---------- Forwarded message --------- > > From: Kevin Liu <[email protected]> > > Date: Thu, Oct 15, 2026 at 6:07 PM > > Subject: GitHub Actions: one user's PR burst on grails-core is saturating > > ubuntu-24.04 for the whole org > > To: <[email protected]> > > > > > > Hi Infra team, > > > > Linux runners in the apache org have been saturated for the past two > days, > > and PR CI across many projects is waiting hours for a runner. > > > > What's happening right now: A single user opened a batch of about 17 PRs > on > > a single project, apache/grails-core, around 13:07 UTC today. Each PR's > CI run > > is using 25–47 jobs at once. Together they hold 569 of the 576 > > ubuntu-24.04 runners > > in use across the org. That's above the 20-job limit in the GitHub > Actions > > policy <https://infra.apache.org/github-actions-policy.html>. > > > > Everyone else on ubuntu-24.04 is stuck behind it, including iceberg, > > zeppelin, datafusion-comet, ozone and iceberg-rust. Snapshot at 16:56 UTC > > across the 345 apache repos pushed to in the last 24 hours: > > Label Running Queued Median wait of queued jobs > > ubuntu-latest 86 1,617 41 min > > ubuntu-24.04 576 1,095 129 min > > ubuntu-26.04 105 439 67 min > > > > Why we care: iceberg-rust's merge queue was timing out on runner waits ( > > apache/iceberg-rust#3442 > > <https://github.com/apache/iceberg-rust/issues/3442>). Yesterday we > moved > > off ubuntu-latest to ubuntu-24.04, which cut our median CI time from 232 > > minutes to 9 minutes. Since about 15:00 UTC today we're back to queuing > for > > over an hour. > > > > Asks: > > > > 1. > > > > Can you reach out to grails-core about capping matrix concurrency > > (strategy.max-parallel: > > 15 or less)? > > 2. > > > > Is the 20-job limit enforced or monitored anywhere? A burst like this > > from one user shouldn't be able to take over the shared pool. > > 3. > > > > Separately, ubuntu-latest has had a backlog of 1,600–2,000 jobs for > two > > days, even while other labels were idle. Is there something specific > to > > that label you can see on your side? > > > > Happy to share the scripts and raw data. > > > > Best, > > > > Kevin > > >
