Hello everyone, I updated the PR-limit analysis with data up to Oct 14 (19 days after the limit): https://gist.github.com/potiuk/a2048d6b5e67c5c2eb07a089c335d02c
* Bulk submissions stay low: PRs from contributors opening more than 5 a week were 17 and 27 in the two full weeks after the limit, compared with 63/week in the 4 weeks before. * We are not scaring people away: 74 and 82 distinct contributors a week, the same as before. * Contributor PRs per day are back to the Jul-Aug level (21.6 vs 21.9). * A correction to what I wrote on Oct 8: the open contributor queue went down to ~590 after the closures, not ~800. It stayed flat until Oct 7, but it has been growing again since then, to 655 on Oct 14. The limit alone does not stop the queue from growing, which is one more reason for the next step. Andrew, fat-catTW - good point about fork CI not matching upstream CI. But it can be addressed by aligning the "push" and "pull_request" runs. There is no difference if CI runs on ASF or "fork" runners; the author of the PR would have to change the workflows in their branch to make it behave differently. Pratiksha - happy to start with an opt-in pilot. It would be too dangerous to enable it for everyone immediately, but we can try with a few volunteers. Unless there are objections, I will start a [LAZY CONSENSUS] on this version (opt-in pilot first) in a few days. J. On Sat, Oct 10, 2026 at 12:56 AM Jarek Potiuk <[email protected]> wrote: > > > The cause is that ci-amd.yml calls many reusable workflows, and each matrix > > sets max-parallel: 20 on its own. Each matrix stays under the limit, but > > they run side by side and add up. GitHub has no setting that caps the whole > > workflow that I'm aware of. > > Yes. GH controls here are heavily limited. But that's even more important to > do "Your PR -> Your CI" - because then the 20 comes not from the limit single > workflow has, but this is maximum runners single **contributor** can run - > which is even better because it also applies to all PRs that contibutor opens. > r > If a single contributor today rebases 5 PRs about the same time and all those > PRs are "significant" - i.e. they trigger the whole matrix of tests, it will > take 15 hours or so to get them all completed if we change approach to "Your > PR -> Your CI" - and they will use 20 runners max. If they do it today where > all runners come from ASF, they can use all the free runners—i.e., it will > take 2 hours—but then it might cause hundreds of jobs running in parallel, > and we cannot limit that. > > That is the tragedy of the commons we experience now in ASF - where > distributing shared resources fairly among many users is very difficult, and > the rules you can set with GH are currently very poor. The rules that Infra > made are simply "poor" approach to control it with whatever is currently > available. > > On the other hand - distributing the CI to be "per user who submits PR" is > **way** more fair - because the user can at most block themselves, not > others. And they must start prioritizing. We are effectively "shifting left" > resource management and get rid of the "commons" part. With this change, the > contributor must start being careful about what they do - because they might > block themselves, rather than others. > > This is really classic "tragedy of the commons" > https://en.wikipedia.org/wiki/Tragedy_of_the_commons - nothing new, - it's a > very well researched economic principle - known and described since Aristotle > time - and absolutely expected for the kind of shared resources (GitHub > Runners time) we share: > > "The tragedy of the commons is the idea that when a resource is shared and > can benefit the individual at a cost to the community, it is rational for the > individual to overuse the resource, even though this collectively leads to > the depletion of the resource for everyone." > > T > > J. > > > > On Fri, Oct 9, 2026 at 4:28 PM Justin Mclean <[email protected]> wrote: >> >> Hi Jarek, >> >> One data point for this thread. Infra’s GitHub Actions policy asks for a cap >> on workflows at 20 concurrent jobs: >> https://infra.apache.org/github-actions-policy.html >> >> The Tests (AMD) workflow goes over it. In its last 20 completed runs it >> peaked at 40 jobs at once, and 12 of the 20 went over 20. I counted only >> jobs that got a runner, so queued jobs that were cancelled are not included. >> In one successful PR run this morning, 30 jobs from 17 test groups were >> running at the same time: >> https://github.com/apache/airflow/actions/runs/37899638118 >> >> The cause is that ci-amd.yml calls many reusable workflows, and each matrix >> sets max-parallel: 20 on its own. Each matrix stays under the limit, but >> they run side by side and add up. GitHub has no setting that caps the whole >> workflow that I'm aware of. >> >> Thanks, >> Justin >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
