Hello everyone, It has been 10 days since we introduced the limit of 5 open PRs for contributors without write access. I want to share early results and propose the next step.
TL;DR * The PR limit is doing what we hoped. The data is early (9 days), and I will redo the analysis in a few weeks. * Proposal: PRs from external contributors run their CI in the contributor's own fork, on GitHub's free runners, instead of on the ASF runners. Nothing changes for committers or for a few stakeholder repositories the PMC decides on. * This is NOT "pay-to-play": CI on public forks is free and has no limit on minutes. The only limit is 20 jobs running at the same time. Small PRs are practically unaffected; big, Airflow-wide PRs take longer to get a CI result. * If there is general consensus, I will start a [LAZY CONSENSUS] next week and start rolling it out right after. I already have a design. 1. Early results of the PR limit Charts and numbers are here: https://gist.github.com/potiuk/a2048d6b5e67c5c2eb07a089c335d02c I compared PRs opened from January to Oct 4, excluding bots and backports. The window after the limit is only 9 days, so these are early observations, not conclusions: * Bulk submissions dropped sharply. PRs from contributors who opened more than 5 PRs in a week went from ~63/week (the 4 weeks before) and ~42/week (Jul-Aug) to 17 in the first week after. That is back to the level of the first half of the year. * We did not scare people away. 74 different contributors opened PRs in the first week after the limit, compared with 82/week just before and 76/week in Jul-Aug. The PRs are spread across more people. * Contributor PRs per day are ~22% lower than in the 4 weeks before, but the same as in the 9 days just before the limit. Part of the decrease happened while we were discussing the limit. * The open queue of contributor PRs went from ~814 to ~590. This is mostly the one-time closure of PRs above the limit. Before that, the queue had grown from ~260 in January to ~870 in early September. It has not grown since (!) * Committers' PRs were not affected. * The share of PRs opened as drafts, which do not count towards the limit, rose slightly (4% -> 7%). Too early to tell, but worth watching. I will re-run the same analysis in 2-3 weeks and share the results. 2. Why a next step is needed AI changed the balance between creating code and reviewing it. Producing a PR now costs almost nothing. Reviewing it costs the same as before - and maintainer time is the one resource we cannot scale. More reviewers would help, but we cannot expect 10x more review time. Today, the whole cost of iterating on a PR is on our side: our review time, and our (ASF-donated) CI capacity, used for every push of every PR. The PR limit caps how many PRs one person can have open. It does not change who carries the cost of getting a PR into a reviewable state. That is what this proposal is about. 3. The proposal: "your PR, your CI" * For contributors without write access, the CI runs in their own fork (on push to the branch), using GitHub's free runners. To get it, a contributor only has to enable GitHub Actions in their fork - once. * The ASF runners keep running CI for: - committers, - a small number of stakeholder repositories the PMC decides on (for example forks/repos used by Astronomer, AWS and Google, which contribute a lot of the code and run shared development), - anyone else the PMC decides to add. For all of them, the experience does not change at all. * The PR shows the result of the CI from the fork, so reviewers see the same green/red status as today. PRs without a CI result get a clear, friendly message explaining what to do. 4. What it means for contributors - iteration speed GitHub's free plan gives public repositories unlimited minutes on standard runners. The limit is concurrency: at most 20 jobs at the same time per account. Our CI uses "selective checks" - it runs only what a change can affect - so the impact depends on the size of the change. I measured real runs from today and replayed them on 20 concurrent runners (these are estimates, slightly optimistic): Type of PR Jobs Job-minutes ASF runners 20 runners UI translations only 33 132 ~49 min ~49 min Single provider (Trino) 50 300 ~31 min ~34 min Core fix (selective) 37 275 ~35 min ~35 min Full matrix (CI change) 138 1307 ~49 min ~85 min+ The full canary run on main (309 jobs) would take ~4h instead of ~2h, but that does not run for contributor PRs. So small, focused PRs - a single provider, docs, UI, a localized core fix - are practically unaffected. Only large PRs that touch "all of Airflow" and need the full matrix will take longer to get a result, roughly 1.5-2x - so it will be slower to iterate on them. I think this is exactly in line with what we want. Big, cross-cutting PRs are also the ones that cost maintainers the most review time. Making their author's iteration a bit slower aligns the incentives: smaller, focused PRs get the fastest feedback, both from CI and from reviewers. It is also the path we already recommend to new contributors. And paired with PR limits - it further limits possibility of one person producing a "mountain of reviews". 5. Next steps * Please share your thoughts in this thread. * If there is a general consensus, I will start a [LAZY CONSENSUS] next week and put it in motion next week as well. I already have a design of how it can be implemented with our current workflows. * More steps will follow. I am using Apache Magpie to iterate on and test some of them, but none of those changes will require anyone to use Magpie. A note about Magpie, because the approach changed based on the feedback we got. Magpie is now focused on individual maintainers, who can experiment and build flows that help them do things faster - without needing project changes and without others having to use it. What benefits the whole community should then become deterministic things: CI, GitHub/repository configuration, documentation - plus analysis of the results, like the one above. This proposal is an example: Magpie helped me prototype and analyze it, but the result is plain GitHub Actions configuration. 6. FAQ - concerns raised so far Q: Isn't this "pay-to-play" / "pay-to-contribute"? A: No. Nobody has to pay anything. CI for public repositories on GitHub's free plan has no minute limit. A contributor gets the same CI, for free, in their fork. The only difference is that at most 20 jobs run at the same time, which affects only large PRs, and only how fast they get a result. Contributors who need more can use runners of their employer, and the PMC can add repositories to the list that uses ASF runners. Q: Will contributors run out of the 2,000 free minutes per month? A: No. That limit applies to private repositories. Public forks of Airflow have no minute limit on standard GitHub runners. Q: How does this reduce the number of PRs? Isn't the real problem that PRs are not reviewed? A: The main goal is not fewer PRs - it is fewer PRs that are not ready to review, and moving the cost of getting a PR green to the author. The review problem is real, and we keep working on it in parallel (review rotation, better triage, surfacing what needs a review). These are complementary, not alternatives. Q: Can we trust the result of CI running in a contributor's fork? They could modify the workflows. A: Contributors can already modify workflow files in their PRs today. We review the code changes, not CI logs, and every merged change runs the full canary build on main. If we want, maintainers can keep an option to run the CI on ASF runners before merging a PR. Q: What about secrets, variables and other integrations? A: None are needed. PRs from forks already run without secrets today, and our CI was designed this way. Q: Will cancelled or missing CI runs confuse contributors? A: Each PR will get a clear message and a link to documentation explaining what to do: enable Actions in your fork, push again. This is a one-time step. Q: Why not ask the ASF for more CI resources, or use donated cloud credits? A: We are doing that too, and it will make CI faster for everyone. But more CI capacity does not create more review time. Free CI for unlimited PRs is part of what makes producing PRs cheaper than reviewing them. Q: Why not run a smaller matrix in PRs (e.g. only the oldest Python)? A: That is a good idea and complementary. Selective checks already do a lot of that, and we can trim further. It makes CI cheaper; it does not change who carries the cost of iterating. Q: Why not require an accepted issue before opening a PR? A: It would move the load from PRs to issues - many agents already open an issue and a PR solving it at the same time - and someone would still need to triage and accept those issues. We can still discuss it separately though - again this is complementary not replacement. Q: Does this affect committers or our regular stakeholders? A: No. Committers and the repositories the PMC adds to the list keep using ASF runners, with no change in experience. Q: Do I need to use Magpie for any of this? A: No. It is plain GitHub Actions and repository configuration. Looking forward to your thoughts. J. -- Drafted with Claude Code (Opus 5); reviewed by Jarek before sending. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
