Hi all, I would like to propose a change to how CI runs for external contributions—specifically, hope that it will help us deal with maintainer overload caused by too many PRs that need our attention.
Basically the idea is to switch where CI runs for non-approved external contributors. Since this affects the contribution experience for everyone who is not a committer or collaborator, I want to discuss it thoroughly before moving forward. I have written this up as AIP-120 on the wiki: https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks Below is a summary of the data and arguments. The Problem (Data from the last 90 days) - 4,009 PRs opened (~45/day). - 65% came from external contributors. - External PRs close without merging at 19.5%, compared to 5.2% for committers and collaborators (~3.8x higher). - That accounts for 508 external PRs in 90 days (~5.6/day) that consumed CI resources and reviewer attention without producing any code changes. CI capacity is limited and costly for the foundation. A full matrix run requires 4,392 job-minutes across 308 jobs, whereas a docs-only change takes 31 job-minutes (a 60x difference). The Proposal 1. External CI in Forks: External contributions run their CI within the contributor's own fork. Project CI is reserved for maintainers and approved organizational stakeholders. 2. Automated Flow: - Workflows run in the contributor's fork on push. - Runs originating from non-approved forks are stopped in apache/airflow. - The PR is automatically set to draft status, labeled, and accompanied by an explanatory comment for the author. - A monitor undrafts the PR once the fork CI turns green. - PRs with no CI activity after a specified period are automatically closed with an explanation (and can be reopened later). 3. Fully Automated: The entire process is deterministic, requires no manual maintainer action, and adds no administrative overhead. Note: Apache SeaTunnel already uses a similar workflow, and several ASF projects do not run PR CI at all. Correction Regarding GitHub Actions Costs I previously suggested this model would work because contributors would spend their own free-tier Actions minutes, creating a financial rate-limit. This was incorrect. GitHub Actions is free and unmetered for public repositories; the 2,000-minute monthly cap applies only to private repos. Because a fork of apache/airflow is public, running CI costs the contributor nothing. The actual limiting factor is concurrency: - Free GitHub accounts are capped at 20 concurrent jobs, whereas our project runs on enterprise-level concurrency. - Docs changes: ~8 minutes in either setup (no difference). - Large core changes: ~88 minutes in a fork vs. ~35 minutes on project infrastructure (repeated on every push). While this creates a time-based gradient rather than a financial one, asking contributors to spend time verifying their own work is a reasonable expectation. It discourages iterating on large, unverified changes directly against project resources while leaving small or documentation contributions largely unaffected. Addressing Objections - "This won't reduce total PRs opened": Correct. The goal is to reduce unready PRs that require maintainer attention. A PR that is automatically drafted, fails to go green, and closes automatically consumes zero reviewer time. - Alternative: Gating PRs on triaged issues: While this has merit in separating feature approval from implementation review, requiring maintainers to approve issues first shifts the burden onto our already large issue backlog rather than eliminating it. (These two approaches are also not mutually exclusive.) Additional Question: Existing Backlog I would also like to discuss whether we should close currently open external PRs when switching to this model, asking authors to reopen them under the new workflow to establish a clean baseline. I view this as a separable proposal. Mass-closing contributor work carries a risk of negative feedback, PR reopen surges, or abandoning valid work. If we choose to do this, messaging must clearly explain that CI capacity is foundation-funded and finite, rather than implying GitHub Actions costs money for the contributor. Discussion Points I am seeking feedback on the general proposal, as well as specific input on: 1. Is a time-based concurrency gate effective enough to pursue? 2. Should we reset or mass-close the existing external PR backlog? 3. How long should a PR sit without CI activity before being automatically closed? 4. What criteria should qualify an organization for a CI exemption? Best regards, Jarek --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
