It's definitely a problem, and getting worse as LLM contributions get more and more common, but I'm not sure this is the fix.
This feels like adding a pretty big barrier of entry for new contributors to a Community Over Code project. If I'm understanding this right, a new contributor is going to submit a PR, have their PR flipped to draft "instantly" and get a message telling them to run the CI in their own fork, go to their fork's settings to enable Actions, and manually sort through which of the 50-ish files in .github/workflows actually need enabling, push again and realize they need a second PR against their own fork to trigger those tests. Create a new PR against their own fork and wait for CI, make the changes, iterate as needed. And all the while there is a countdown timer threatening to close my PR if I don't get this sorted out. That all feels like a pretty horrible experience, but maybe I'm making more of it than it really is in practice. I'm not sure about the "A monitor undrafts the PR once the fork CI turns green." part. It feels pretty hand-wavey for what will turn out to be a critical part of the plan. Did you have something in mind for how we can monitor an external fork's CI and modify the state of the PR in our repo based on that? I'm also not sure about some of your numbers. My PRs routinely sit in CI for an hour right now, which is longer than the upper end of your estimates and a spot check of 10 random open PRs shows they all take about an hour, so it's not just me. We also already have to manually approve the CI run for all first-time contributors as Ash mentioned. A quick Claude search looks like there are 1,300 runs sitting in action_required, which is several hundred PRs whose CI never ran and therefore cost nothing. On the flip side, I think the math is off on the docs-to-full-matrix spread: 4,392 ÷ 31 is about 142x, not 60x. 60x looks like the ratio against the largest sampled PR run rather than the full matrix, so I think there is actually a bigger difference than you gave credit for. - ferruzzi ________________________________ From: Jarek Potiuk <[email protected]> Sent: Friday, September 18, 2026 12:51 PM To: [email protected] <[email protected]> Subject: RE: [EXT] [DISCUSS] Run CI for external contributions in contributor forks CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Ash: > How many of these PRs are for new contributors where we need to approve running CI already? The detailed stats are in the AIP - > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks - less then 7% Jyun, > Authors should be given an explicit window to update their PRs before any automated closure takes place. Update how? J On Fri, Sep 18, 2026 at 9:11 PM Jyun-An Chen <[email protected]> wrote: > Regarding the existing external PR backlog: > > I have an idea of applying the new CI policy only to new PRs first, while > setting a sunset date (e.g., 30 days) for the existing PR backlog. Authors > should be given an explicit window to update their PRs before any automated > closure takes place. > > Colten > > Ash Berlin-Taylor <[email protected]> 於 2026年9月19日週六 上午1:58寫道： > > > How many of these PRs are for new contributors where we need to approve > > running CI already? > > > > > On 18 Sep 2026, at 15:24, Jarek Potiuk <[email protected]> wrote: > > > > > > Hi all, > > > > > > I would like to propose a change to how CI runs for external > > > contributions—specifically, > > > hope that it will help us deal with maintainer overload caused by too > > many PRs > > > that need our attention. > > > > > > Basically the idea is to switch where CI runs for non-approved > > > external contributors. > > > > > > Since this affects the contribution experience for everyone who is not > > > a committer or collaborator, I want to discuss it thoroughly before > > > moving forward. > > > > > > I have written this up as AIP-120 on the wiki: > > > > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks > > > > > > Below is a summary of the data and arguments. > > > > > > The Problem (Data from the last 90 days) > > > > > > - 4,009 PRs opened (~45/day). > > > - 65% came from external contributors. > > > - External PRs close without merging at 19.5%, compared to 5.2% for > > > committers and collaborators (~3.8x higher). > > > - That accounts for 508 external PRs in 90 days (~5.6/day) that > > > consumed CI resources and reviewer attention without producing any > > > code changes. > > > > > > CI capacity is limited and costly for the foundation. A full matrix > > > run requires 4,392 job-minutes across 308 jobs, whereas a docs-only > > > change takes 31 job-minutes (a 60x difference). > > > > > > The Proposal > > > > > > 1. External CI in Forks: External contributions run their CI within > > > the contributor's own fork. Project CI is reserved for maintainers and > > > approved organizational stakeholders. > > > > > > 2. Automated Flow: > > > - Workflows run in the contributor's fork on push. > > > - Runs originating from non-approved forks are stopped in > > apache/airflow. > > > - The PR is automatically set to draft status, labeled, and > > > accompanied by an explanatory comment for the author. > > > - A monitor undrafts the PR once the fork CI turns green. > > > - PRs with no CI activity after a specified period are > > > automatically closed with an explanation (and can be reopened later). > > > > > > 3. Fully Automated: The entire process is deterministic, requires no > > > manual maintainer action, and adds no administrative overhead. > > > > > > Note: Apache SeaTunnel already uses a similar workflow, and several > > > ASF projects do not run PR CI at all. > > > > > > Correction Regarding GitHub Actions Costs > > > > > > I previously suggested this model would work because contributors > > > would spend their own free-tier Actions minutes, creating a financial > > > rate-limit. This was incorrect. > > > > > > GitHub Actions is free and unmetered for public repositories; the > > > 2,000-minute monthly cap applies only to private repos. Because a fork > > > of apache/airflow is public, running CI costs the contributor nothing. > > > > > > The actual limiting factor is concurrency: > > > > > > - Free GitHub accounts are capped at 20 concurrent jobs, whereas our > > > project runs on enterprise-level concurrency. > > > - Docs changes: ~8 minutes in either setup (no difference). > > > - Large core changes: ~88 minutes in a fork vs. ~35 minutes on > > > project infrastructure (repeated on every push). > > > > > > While this creates a time-based gradient rather than a financial one, > > > asking contributors to spend time verifying their own work is a > > > reasonable expectation. It discourages iterating on large, unverified > > > changes directly against project resources while leaving small or > > > documentation contributions largely unaffected. > > > > > > Addressing Objections > > > > > > - "This won't reduce total PRs opened": Correct. The goal is to > > > reduce unready PRs that require maintainer attention. A PR that is > > > automatically drafted, fails to go green, and closes automatically > > > consumes zero reviewer time. > > > - Alternative: Gating PRs on triaged issues: While this has merit in > > > separating feature approval from implementation review, requiring > > > maintainers to approve issues first shifts the burden onto our already > > > large issue backlog rather than eliminating it. (These two approaches > > > are also not mutually exclusive.) > > > > > > Additional Question: Existing Backlog > > > > > > I would also like to discuss whether we should close currently open > > > external PRs when switching to this model, asking authors to reopen > > > them under the new workflow to establish a clean baseline. > > > > > > I view this as a separable proposal. Mass-closing contributor work > > > carries a risk of negative feedback, PR reopen surges, or abandoning > > > valid work. If we choose to do this, messaging must clearly explain > > > that CI capacity is foundation-funded and finite, rather than implying > > > GitHub Actions costs money for the contributor. > > > > > > Discussion Points > > > > > > I am seeking feedback on the general proposal, as well as specific > input > > on: > > > > > > 1. Is a time-based concurrency gate effective enough to pursue? > > > 2. Should we reset or mass-close the existing external PR backlog? > > > 3. How long should a PR sit without CI activity before being > > > automatically closed? > > > 4. What criteria should qualify an organization for a CI exemption? > > > > > > Best regards, > > > Jarek > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
