Hi, I found the new CPython AI usage guidelines [1] interesting. Having one label to request the author of the PR to explain the contribution in their own words (no AI usage) sounds like an interesting option. A second label for expected AI usage not met is used there to close the PR.
Regards, Raúl [1] https://discuss.python.org/t/updated-triage-guidelines-and-process-for-our-ai-policy/109717 El mar, 27 oct 2026 a las 16:15, Nic Crane (<[email protected]>) escribió: > > Hi folks, > > I spent some time today having a look at the impact of AI on the project, > and also had AI (ironically) deep dive into some of this[1], looking at a > few metrics around reviewer capacity, examining PR discussions and also > looking at discussions here on the developer mailing list, with some input > from me in there as well. > > I've summarised the main problems and some potential solutions below, but > I'd like to hear from other maintainers on whether this broadly reflects > your experiences? > > Thanks, > > Nic > > > Main problems currently: > - increased maintainer time spent reviewing PRs which are ultimately > abandoned after time spent reviewing, or are predominantly AI-generated so > take longer to merge as they need more feedback > - maintainer time spend enforcing rules around AI-generated PRs, and > non-committers reviewing PRs with little ability to flag them as problematic > - sheer volume of AI-generated PRs, as well as around a third of recent PRs > skipping AI disclosure at all > > Recommendations: > 1. Move the PR cap to .asf.yaml creation_cap before pull_request_target is > disabled in November (#55164). (awaiting merge of > https://github.com/apache/infrastructure-asfyaml/pull/135) > 2. Make AI use disclosure mandatory, as numpy, pandas and matplotlib do, > with a bot that converts PRs missing it to draft. > 3. Adopt #55260’s rule against AI-written review replies, and rok’s neutral > needs-author-engagement label rather than an AI label. > 4. Auto-draft PRs idle for 14 days with conflicts, failing CI or changes > requested, and trial fork-hosted CI for non-committer PRs as DataFusion and > arrow-rs do. > 5. Gate PR opening on triage state as pandas does, so PRs are only expected > against issues a maintainer has confirmed. > 6. Track monthly: new-contributor PRs with no human response, and reviewer > messages on PRs that never merge. > 7. Grow the merge group from the people already triaging: give the active > non-committer reviewers the collaborator role and a label-triggered > autoclose, and treat that as the path to committer. This is the only > measure that adds capacity rather than cutting inflow. > > [1] Full report here: > https://thisisnic.github.io/ai-arrow-impact/report.html > > On Mon, 26 Oct 2026 at 12:30, Raúl Cumplido <[email protected]> wrote: > > > Hi, > > > > I've followed up with a PR [1] based on the comments from Pedro Matias > > and Rok so we have this documented. See that PR for the documentation > > changes. > > > > Regards, > > Raúl > > > > [1] https://github.com/apache/arrow/pull/55260 > > > > El mar, 22 sept 2026 a las 10:57, Jean-Baptiste Onofré > > (<[email protected]>) escribió: > > > > > > Discussions must take place publicly, even between contributors from the > > > same company. > > > > > > At the ASF, we contribute as individuals, so company affiliation does not > > > matter here; we are all project contributors. > > > > > > All discussions and decisions should happen on the dev mailing list. > > > Discussions regarding quality and implementation should also take place > > > publicly on the relevant PR or the dev mailing list. > > > > > > Regards, > > > JB > > > > > > > > > On Mon, Sep 21, 2026 at 8:04 PM Dmitry Chirkov <[email protected]> > > wrote: > > > > > > > I like Rok's proposed approach. > > > > > > > > Pedro and I discussed this topic offline. While we can have private, > > > > internal conversations with authors regarding contribution quality > > within > > > > the company, open source contributions lack direct messaging > > capabilities. > > > > As a result, this interaction - unfortunately - needs to take place > > > > publicly. > > > > > > > > Dmitry Chirkov, > > > > Dremio > > > > > > > > On Mon, Sep 21, 2026 at 9:40 AM Rok Mihevc <[email protected]> > > wrote: > > > > > > > > > Agreed with Pedro's proposal as well! > > > > > > > > > > I'd suggest another item here: let's introduce a GitHub label: > > > > > "needs-author-engagement". Reviewer could use this label to flag PRs > > that > > > > > require substantial responses from their authors before a productive > > > > review > > > > > can continue. This label should not focus on AI use, but rather on > > the > > > > > author's observable understanding of the PR's subject matter and > > their > > > > > engagement with reviewer feedback. Such label would also help other > > > > > reviewers avoid duplicate effort and serve as feedback to the author > > of > > > > the > > > > > PR. > > > > > > > > > > Rok > > > > > > > > > > On Mon, Sep 21, 2026 at 6:24 PM Nic Crane <[email protected]> > > wrote: > > > > > > > > > > > Hi Pedro, > > > > > > > > > > > > Thanks for raising this issue, and I think it's very timely in that > > > > many > > > > > of > > > > > > us are experiencing similar issues. Your proposal sounds good to > > me. > > > > > > > > > > > > Nic > > > > > > > > > > > > On Fri, 18 Sept 2026 at 10:33, Pedro Matias < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > I'd like to revive this thread. My goal is simple: I want to > > discuss > > > > > > > establishing guidelines for reviewers, especially those of us > > who are > > > > > not > > > > > > > Arrow committers/PMCs, on how to handle low-quality PRs with weak > > > > > > > engagement. The community added AI guidelines for contributors, > > but > > > > not > > > > > > for > > > > > > > reviewers: > > https://arrow.apache.org/docs/developers/reviewing.html > > > > > > > > > > > > > > Lately I've been reviewing a PR that I think fits this > > description. > > > > > I've > > > > > > > pointed the person to the AI guidelines at > > > > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > > , > > > > > > > but the guidelines were ignored. I can continue steering the code > > > > with > > > > > > > reviews, but I am afraid this might encourage others to repeat > > this > > > > > > pattern > > > > > > > of weak engagement. > > > > > > > > > > > > > > I repeat R. Tyler Croy's ask: "don't rely on everybody following > > the > > > > > > rules, > > > > > > > and come up with an agreed upon way to handle those that don't." > > > > > > > > > > > > > > I looked into other projects to see if any of them have something > > > > > > similar. > > > > > > > LLVM has guidance [0] on both how to warn the contributor and > > when to > > > > > > > escalate to someone with permission to lock the conversation, as > > well > > > > > as > > > > > > a > > > > > > > label that can be added to a low effort PR. > > > > > > > > > > > > > > I propose we add a section titled "Handling violations of the AI > > > > > > > contribution guidelines" to the reviewer guidelines. I wrote a > > small > > > > > > draft > > > > > > > [1] of what it could look like. I'm happy to iterate on it if > > people > > > > > > think > > > > > > > this is something worth adding. > > > > > > > > > > > > > > I'm particularly curious about what committers/PMC members think > > > > should > > > > > > be > > > > > > > the way for read-only reviewers to escalate to maintainers. The > > > > current > > > > > > > proposal suggests pinging someone, which might be too noisy. > > > > > > > > > > > > > > 0- https://llvm.org/docs/AIToolPolicy.html#handling-violations > > > > > > > > > > > > > > 1- When a reviewer finds that a contribution does not seem to > > conform > > > > > to > > > > > > > the guidelines for AI usage, they should respond with the > > following > > > > > > > message: > > > > > > > " > > > > > > > This PR does not seem to meet the standards for AI generated > > > > > > contributions. > > > > > > > Please read the guidelines at > > > > > > > > > > > > > > https://arrow.apache.org/docs/developers/overview.html#ai-generated-code > > > > > > > and ensure you modify your PR to conform to the rules. > > > > > > > " > > > > > > > If the contributor fails to adapt their work and/or engagement > > level > > > > to > > > > > > > meet the guidelines' standards, maintainers may close the PR. > > > > Reviewers > > > > > > > without permission to close the PR should escalate by pinging a > > > > > > maintainer > > > > > > > via comment indicating that they do not believe the change meets > > the > > > > > > > standards. > > > > > > > > > > > > > > Best regards, > > > > > > > Pedro Matias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 13, 2026 at 3:53 PM Nic Crane <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > On a similar note, after conversations with folks around what > > > > appear > > > > > to > > > > > > > be > > > > > > > > AI-generated mailing list responses, I've also opened a PR > > > > suggesting > > > > > > > > people disclose any AI-generated questions they post to mailing > > > > list > > > > > > > > discussions; feel free to add any comments there (if you're a > > > > human! > > > > > > ;) ) > > > > > > > > > > > > > > > > https://github.com/apache/arrow/pull/49277/changes > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 20:42, Nic Crane <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > PR here for anyone interested: > > > > > > > > https://github.com/apache/arrow/pull/48952 > > > > > > > > > > > > > > > > > > On Thu, 22 Jan 2026 at 09:56, Nic Crane <[email protected] > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> Thanks Andrew, I really like how you spell out the reasoning > > > > > around > > > > > > > it, > > > > > > > > I > > > > > > > > >> will see how we can incorporate some of those ideas > > > > > > > > >> > > > > > > > > >> On Thu, 22 Jan 2026 at 09:23, Andrew Lamb < > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> > We have had repeated attempts at contributions by some > > folks > > > > > who > > > > > > > > simply > > > > > > > > >>> do not understand their generated code and when asked for > > > > > > > > clarification, > > > > > > > > >>> have the LLM generate more incorrect commentary. It's very > > > > > > > > >>> Dunning-Krueger > > > > > > > > >>> and leads to lots of frustration all around. > > > > > > > > >>> > > > > > > > > >>> We saw this too in DataFusion and I was pleased with what > > we > > > > came > > > > > > up > > > > > > > > with > > > > > > > > >>> for rationale about why it is not helpful[1]. Basically the > > > > > > reviewers > > > > > > > > are > > > > > > > > >>> more efficient using the LLM tools directly and the > > contributor > > > > > > isn't > > > > > > > > >>> learning anything either. > > > > > > > > >>> > > > > > > > > >>> Andrew > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> [1]: > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://datafusion.apache.org/contributor-guide/index.html#why-fully-ai-generated-prs-without-understanding-are-not-helpful > > > > > > > > >>> > > > > > > > > >>> On Mon, Jan 19, 2026 at 12:48 PM R Tyler Croy < > > > > > [email protected]> > > > > > > > > >>> wrote: > > > > > > > > >>> > > > > > > > > >>> > (replies inline) > > > > > > > > >>> > > > > > > > > > >>> > On Sunday, January 18th, 2026 at 7:43 PM, Gang Wu < > > > > > > > [email protected]> > > > > > > > > >>> > wrote: > > > > > > > > >>> > > > > > > > > > >>> > > - Summitters should review all lines of generated code > > > > before > > > > > > > > >>> creating > > > > > > > > >>> > the > > > > > > > > >>> > > PR to > > > > > > > > >>> > > understand every piece of detail just like they are > > written > > > > > by > > > > > > > the > > > > > > > > >>> > > submitters > > > > > > > > >>> > > themselves. > > > > > > > > >>> > > - AI tools are notorious for generating overly verbose > > > > > > comments, > > > > > > > > >>> > unnecessary > > > > > > > > >>> > > test cases, fixing test failures using wrong > > approaches, > > > > etc. > > > > > > > Make > > > > > > > > >>> sure > > > > > > > > >>> > > these > > > > > > > > >>> > > are checked and fixed. > > > > > > > > >>> > > - Reviewers are humans, so please try to break down > > large > > > > PRs > > > > > > > into > > > > > > > > >>> > smaller > > > > > > > > >>> > > ones to make reviewers' life easier to get PRs promptly > > > > > > reviewed. > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > Like others I think Nic's draft is a good one, I would > > like > > > > to > > > > > > > offer > > > > > > > > >>> some > > > > > > > > >>> > thoughts as a maintainer (delta-rs) which has received > > > > > increased > > > > > > > > >>> > AI-assisted pull requests over the past six months. > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > The "PR may be closed without further review" statement I > > > > would > > > > > > > > >>> strongly > > > > > > > > >>> > encourage moving to the very beginning of the policy. I > > > > would > > > > > > also > > > > > > > > >>> > encourage labels being used like "ai-assisted" to signal > > to > > > > > other > > > > > > > > >>> > contributors who may or may not wish to engage in > > reviewing > > > > > > > potential > > > > > > > > >>> slop. > > > > > > > > >>> > > > > > > > > > >>> > We have had repeated attempts at contributions by some > > folks > > > > > who > > > > > > > > >>> simply do > > > > > > > > >>> > not understand their generated code and when asked for > > > > > > > clarification, > > > > > > > > >>> have > > > > > > > > >>> > the LLM generate more incorrect commentary. It's very > > > > > > > > Dunning-Krueger > > > > > > > > >>> and > > > > > > > > >>> > leads to lots of frustration all around. > > > > > > > > >>> > > > > > > > > > >>> > Like most policies it's important to speak to those that > > are > > > > > > acting > > > > > > > > in > > > > > > > > >>> > good faith but don't rely on everybody following the > > rules, > > > > and > > > > > > > come > > > > > > > > up > > > > > > > > >>> > with an agreed upon way to handle those that don't. > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > Either way I think it's good to ship! :) > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > Cheers > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
