Hi folks, I spent some time today having a look at the impact of AI on the project, and also had AI (ironically) deep dive into some of this[1], looking at a few metrics around reviewer capacity, examining PR discussions and also looking at discussions here on the developer mailing list, with some input from me in there as well.
I've summarised the main problems and some potential solutions below, but I'd like to hear from other maintainers on whether this broadly reflects your experiences? Thanks, Nic Main problems currently: - increased maintainer time spent reviewing PRs which are ultimately abandoned after time spent reviewing, or are predominantly AI-generated so take longer to merge as they need more feedback - maintainer time spend enforcing rules around AI-generated PRs, and non-committers reviewing PRs with little ability to flag them as problematic - sheer volume of AI-generated PRs, as well as around a third of recent PRs skipping AI disclosure at all Recommendations: 1. Move the PR cap to .asf.yaml creation_cap before pull_request_target is disabled in November (#55164). (awaiting merge of https://github.com/apache/infrastructure-asfyaml/pull/135) 2. Make AI use disclosure mandatory, as numpy, pandas and matplotlib do, with a bot that converts PRs missing it to draft. 3. Adopt #55260’s rule against AI-written review replies, and rok’s neutral needs-author-engagement label rather than an AI label. 4. Auto-draft PRs idle for 14 days with conflicts, failing CI or changes requested, and trial fork-hosted CI for non-committer PRs as DataFusion and arrow-rs do. 5. Gate PR opening on triage state as pandas does, so PRs are only expected against issues a maintainer has confirmed. 6. Track monthly: new-contributor PRs with no human response, and reviewer messages on PRs that never merge. 7. Grow the merge group from the people already triaging: give the active non-committer reviewers the collaborator role and a label-triggered autoclose, and treat that as the path to committer. This is the only measure that adds capacity rather than cutting inflow. [1] Full report here: https://thisisnic.github.io/ai-arrow-impact/report.html On Mon, 26 Oct 2026 at 12:30, Raúl Cumplido <[email protected]> wrote: > Hi, > > I've followed up with a PR [1] based on the comments from Pedro Matias > and Rok so we have this documented. See that PR for the documentation > changes. > > Regards, > Raúl > > [1] https://github.com/apache/arrow/pull/55260 > > El mar, 22 sept 2026 a las 10:57, Jean-Baptiste Onofré > (<[email protected]>) escribió: > > > > Discussions must take place publicly, even between contributors from the > > same company. > > > > At the ASF, we contribute as individuals, so company affiliation does not > > matter here; we are all project contributors. > > > > All discussions and decisions should happen on the dev mailing list. > > Discussions regarding quality and implementation should also take place > > publicly on the relevant PR or the dev mailing list. > > > > Regards, > > JB > > > > > > On Mon, Sep 21, 2026 at 8:04 PM Dmitry Chirkov <[email protected]> > wrote: > > > > > I like Rok's proposed approach. > > > > > > Pedro and I discussed this topic offline. While we can have private, > > > internal conversations with authors regarding contribution quality > within > > > the company, open source contributions lack direct messaging > capabilities. > > > As a result, this interaction - unfortunately - needs to take place > > > publicly. > > > > > > Dmitry Chirkov, > > > Dremio > > > > > > On Mon, Sep 21, 2026 at 9:40 AM Rok Mihevc <[email protected]> > wrote: > > > > > > > Agreed with Pedro's proposal as well! > > > > > > > > I'd suggest another item here: let's introduce a GitHub label: > > > > "needs-author-engagement". Reviewer could use this label to flag PRs > that > > > > require substantial responses from their authors before a productive > > > review > > > > can continue. This label should not focus on AI use, but rather on > the > > > > author's observable understanding of the PR's subject matter and > their > > > > engagement with reviewer feedback. Such label would also help other > > > > reviewers avoid duplicate effort and serve as feedback to the author > of > > > the > > > > PR. > > > > > > > > Rok > > > > > > > > On Mon, Sep 21, 2026 at 6:24 PM Nic Crane <[email protected]> > wrote: > > > > > > > > > Hi Pedro, > > > > > > > > > > Thanks for raising this issue, and I think it's very timely in that > > > many > > > > of > > > > > us are experiencing similar issues. Your proposal sounds good to > me. > > > > > > > > > > Nic > > > > > > > > > > On Fri, 18 Sept 2026 at 10:33, Pedro Matias < > [email protected]> > > > > > wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > I'd like to revive this thread. My goal is simple: I want to > discuss > > > > > > establishing guidelines for reviewers, especially those of us > who are > > > > not > > > > > > Arrow committers/PMCs, on how to handle low-quality PRs with weak > > > > > > engagement. The community added AI guidelines for contributors, > but > > > not > > > > > for > > > > > > reviewers: > https://arrow.apache.org/docs/developers/reviewing.html > > > > > > > > > > > > Lately I've been reviewing a PR that I think fits this > description. > > > > I've > > > > > > pointed the person to the AI guidelines at > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > , > > > > > > but the guidelines were ignored. I can continue steering the code > > > with > > > > > > reviews, but I am afraid this might encourage others to repeat > this > > > > > pattern > > > > > > of weak engagement. > > > > > > > > > > > > I repeat R. Tyler Croy's ask: "don't rely on everybody following > the > > > > > rules, > > > > > > and come up with an agreed upon way to handle those that don't." > > > > > > > > > > > > I looked into other projects to see if any of them have something > > > > > similar. > > > > > > LLVM has guidance [0] on both how to warn the contributor and > when to > > > > > > escalate to someone with permission to lock the conversation, as > well > > > > as > > > > > a > > > > > > label that can be added to a low effort PR. > > > > > > > > > > > > I propose we add a section titled "Handling violations of the AI > > > > > > contribution guidelines" to the reviewer guidelines. I wrote a > small > > > > > draft > > > > > > [1] of what it could look like. I'm happy to iterate on it if > people > > > > > think > > > > > > this is something worth adding. > > > > > > > > > > > > I'm particularly curious about what committers/PMC members think > > > should > > > > > be > > > > > > the way for read-only reviewers to escalate to maintainers. The > > > current > > > > > > proposal suggests pinging someone, which might be too noisy. > > > > > > > > > > > > 0- https://llvm.org/docs/AIToolPolicy.html#handling-violations > > > > > > > > > > > > 1- When a reviewer finds that a contribution does not seem to > conform > > > > to > > > > > > the guidelines for AI usage, they should respond with the > following > > > > > > message: > > > > > > " > > > > > > This PR does not seem to meet the standards for AI generated > > > > > contributions. > > > > > > Please read the guidelines at > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > > and ensure you modify your PR to conform to the rules. > > > > > > " > > > > > > If the contributor fails to adapt their work and/or engagement > level > > > to > > > > > > meet the guidelines' standards, maintainers may close the PR. > > > Reviewers > > > > > > without permission to close the PR should escalate by pinging a > > > > > maintainer > > > > > > via comment indicating that they do not believe the change meets > the > > > > > > standards. > > > > > > > > > > > > Best regards, > > > > > > Pedro Matias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 13, 2026 at 3:53 PM Nic Crane <[email protected]> > > > wrote: > > > > > > > > > > > > > On a similar note, after conversations with folks around what > > > appear > > > > to > > > > > > be > > > > > > > AI-generated mailing list responses, I've also opened a PR > > > suggesting > > > > > > > people disclose any AI-generated questions they post to mailing > > > list > > > > > > > discussions; feel free to add any comments there (if you're a > > > human! > > > > > ;) ) > > > > > > > > > > > > > > https://github.com/apache/arrow/pull/49277/changes > > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 20:42, Nic Crane <[email protected]> > > > wrote: > > > > > > > > > > > > > > > PR here for anyone interested: > > > > > > > https://github.com/apache/arrow/pull/48952 > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 09:56, Nic Crane <[email protected] > > > > > > wrote: > > > > > > > > > > > > > > > >> Thanks Andrew, I really like how you spell out the reasoning > > > > around > > > > > > it, > > > > > > > I > > > > > > > >> will see how we can incorporate some of those ideas > > > > > > > >> > > > > > > > >> On Thu, 22 Jan 2026 at 09:23, Andrew Lamb < > [email protected] > > > > > > > > > > wrote: > > > > > > > >> > > > > > > > >>> > We have had repeated attempts at contributions by some > folks > > > > who > > > > > > > simply > > > > > > > >>> do not understand their generated code and when asked for > > > > > > > clarification, > > > > > > > >>> have the LLM generate more incorrect commentary. It's very > > > > > > > >>> Dunning-Krueger > > > > > > > >>> and leads to lots of frustration all around. > > > > > > > >>> > > > > > > > >>> We saw this too in DataFusion and I was pleased with what > we > > > came > > > > > up > > > > > > > with > > > > > > > >>> for rationale about why it is not helpful[1]. Basically the > > > > > reviewers > > > > > > > are > > > > > > > >>> more efficient using the LLM tools directly and the > contributor > > > > > isn't > > > > > > > >>> learning anything either. > > > > > > > >>> > > > > > > > >>> Andrew > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> [1]: > > > > > > > >>> > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > https://datafusion.apache.org/contributor-guide/index.html#why-fully-ai-generated-prs-without-understanding-are-not-helpful > > > > > > > >>> > > > > > > > >>> On Mon, Jan 19, 2026 at 12:48 PM R Tyler Croy < > > > > [email protected]> > > > > > > > >>> wrote: > > > > > > > >>> > > > > > > > >>> > (replies inline) > > > > > > > >>> > > > > > > > > >>> > On Sunday, January 18th, 2026 at 7:43 PM, Gang Wu < > > > > > > [email protected]> > > > > > > > >>> > wrote: > > > > > > > >>> > > > > > > > > >>> > > - Summitters should review all lines of generated code > > > before > > > > > > > >>> creating > > > > > > > >>> > the > > > > > > > >>> > > PR to > > > > > > > >>> > > understand every piece of detail just like they are > written > > > > by > > > > > > the > > > > > > > >>> > > submitters > > > > > > > >>> > > themselves. > > > > > > > >>> > > - AI tools are notorious for generating overly verbose > > > > > comments, > > > > > > > >>> > unnecessary > > > > > > > >>> > > test cases, fixing test failures using wrong > approaches, > > > etc. > > > > > > Make > > > > > > > >>> sure > > > > > > > >>> > > these > > > > > > > >>> > > are checked and fixed. > > > > > > > >>> > > - Reviewers are humans, so please try to break down > large > > > PRs > > > > > > into > > > > > > > >>> > smaller > > > > > > > >>> > > ones to make reviewers' life easier to get PRs promptly > > > > > reviewed. > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > Like others I think Nic's draft is a good one, I would > like > > > to > > > > > > offer > > > > > > > >>> some > > > > > > > >>> > thoughts as a maintainer (delta-rs) which has received > > > > increased > > > > > > > >>> > AI-assisted pull requests over the past six months. > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > The "PR may be closed without further review" statement I > > > would > > > > > > > >>> strongly > > > > > > > >>> > encourage moving to the very beginning of the policy. I > > > would > > > > > also > > > > > > > >>> > encourage labels being used like "ai-assisted" to signal > to > > > > other > > > > > > > >>> > contributors who may or may not wish to engage in > reviewing > > > > > > potential > > > > > > > >>> slop. > > > > > > > >>> > > > > > > > > >>> > We have had repeated attempts at contributions by some > folks > > > > who > > > > > > > >>> simply do > > > > > > > >>> > not understand their generated code and when asked for > > > > > > clarification, > > > > > > > >>> have > > > > > > > >>> > the LLM generate more incorrect commentary. It's very > > > > > > > Dunning-Krueger > > > > > > > >>> and > > > > > > > >>> > leads to lots of frustration all around. > > > > > > > >>> > > > > > > > > >>> > Like most policies it's important to speak to those that > are > > > > > acting > > > > > > > in > > > > > > > >>> > good faith but don't rely on everybody following the > rules, > > > and > > > > > > come > > > > > > > up > > > > > > > >>> > with an agreed upon way to handle those that don't. > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > Either way I think it's good to ship! :) > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > Cheers > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >
