Hello all, I'd like to revive this thread. My goal is simple: I want to discuss establishing guidelines for reviewers, especially those of us who are not Arrow committers/PMCs, on how to handle low-quality PRs with weak engagement. The community added AI guidelines for contributors, but not for reviewers: https://arrow.apache.org/docs/developers/reviewing.html
Lately I've been reviewing a PR that I think fits this description. I've pointed the person to the AI guidelines at https://arrow.apache.org/docs/developers/overview.html#ai-generated-code, but the guidelines were ignored. I can continue steering the code with reviews, but I am afraid this might encourage others to repeat this pattern of weak engagement. I repeat R. Tyler Croy's ask: "don't rely on everybody following the rules, and come up with an agreed upon way to handle those that don't." I looked into other projects to see if any of them have something similar. LLVM has guidance [0] on both how to warn the contributor and when to escalate to someone with permission to lock the conversation, as well as a label that can be added to a low effort PR. I propose we add a section titled "Handling violations of the AI contribution guidelines" to the reviewer guidelines. I wrote a small draft [1] of what it could look like. I'm happy to iterate on it if people think this is something worth adding. I'm particularly curious about what committers/PMC members think should be the way for read-only reviewers to escalate to maintainers. The current proposal suggests pinging someone, which might be too noisy. 0- https://llvm.org/docs/AIToolPolicy.html#handling-violations 1- When a reviewer finds that a contribution does not seem to conform to the guidelines for AI usage, they should respond with the following message: " This PR does not seem to meet the standards for AI generated contributions. Please read the guidelines at https://arrow.apache.org/docs/developers/overview.html#ai-generated-code and ensure you modify your PR to conform to the rules. " If the contributor fails to adapt their work and/or engagement level to meet the guidelines' standards, maintainers may close the PR. Reviewers without permission to close the PR should escalate by pinging a maintainer via comment indicating that they do not believe the change meets the standards. Best regards, Pedro Matias On Fri, Feb 13, 2026 at 3:53 PM Nic Crane <[email protected]> wrote: > On a similar note, after conversations with folks around what appear to be > AI-generated mailing list responses, I've also opened a PR suggesting > people disclose any AI-generated questions they post to mailing list > discussions; feel free to add any comments there (if you're a human! ;) ) > > https://github.com/apache/arrow/pull/49277/changes > > > On Thu, 22 Jan 2026 at 20:42, Nic Crane <[email protected]> wrote: > > > PR here for anyone interested: > https://github.com/apache/arrow/pull/48952 > > > > On Thu, 22 Jan 2026 at 09:56, Nic Crane <[email protected]> wrote: > > > >> Thanks Andrew, I really like how you spell out the reasoning around it, > I > >> will see how we can incorporate some of those ideas > >> > >> On Thu, 22 Jan 2026 at 09:23, Andrew Lamb <[email protected]> wrote: > >> > >>> > We have had repeated attempts at contributions by some folks who > simply > >>> do not understand their generated code and when asked for > clarification, > >>> have the LLM generate more incorrect commentary. It's very > >>> Dunning-Krueger > >>> and leads to lots of frustration all around. > >>> > >>> We saw this too in DataFusion and I was pleased with what we came up > with > >>> for rationale about why it is not helpful[1]. Basically the reviewers > are > >>> more efficient using the LLM tools directly and the contributor isn't > >>> learning anything either. > >>> > >>> Andrew > >>> > >>> > >>> [1]: > >>> > >>> > https://datafusion.apache.org/contributor-guide/index.html#why-fully-ai-generated-prs-without-understanding-are-not-helpful > >>> > >>> On Mon, Jan 19, 2026 at 12:48 PM R Tyler Croy <[email protected]> > >>> wrote: > >>> > >>> > (replies inline) > >>> > > >>> > On Sunday, January 18th, 2026 at 7:43 PM, Gang Wu <[email protected]> > >>> > wrote: > >>> > > >>> > > - Summitters should review all lines of generated code before > >>> creating > >>> > the > >>> > > PR to > >>> > > understand every piece of detail just like they are written by the > >>> > > submitters > >>> > > themselves. > >>> > > - AI tools are notorious for generating overly verbose comments, > >>> > unnecessary > >>> > > test cases, fixing test failures using wrong approaches, etc. Make > >>> sure > >>> > > these > >>> > > are checked and fixed. > >>> > > - Reviewers are humans, so please try to break down large PRs into > >>> > smaller > >>> > > ones to make reviewers' life easier to get PRs promptly reviewed. > >>> > > >>> > > >>> > Like others I think Nic's draft is a good one, I would like to offer > >>> some > >>> > thoughts as a maintainer (delta-rs) which has received increased > >>> > AI-assisted pull requests over the past six months. > >>> > > >>> > > >>> > The "PR may be closed without further review" statement I would > >>> strongly > >>> > encourage moving to the very beginning of the policy. I would also > >>> > encourage labels being used like "ai-assisted" to signal to other > >>> > contributors who may or may not wish to engage in reviewing potential > >>> slop. > >>> > > >>> > We have had repeated attempts at contributions by some folks who > >>> simply do > >>> > not understand their generated code and when asked for clarification, > >>> have > >>> > the LLM generate more incorrect commentary. It's very > Dunning-Krueger > >>> and > >>> > leads to lots of frustration all around. > >>> > > >>> > Like most policies it's important to speak to those that are acting > in > >>> > good faith but don't rely on everybody following the rules, and come > up > >>> > with an agreed upon way to handle those that don't. > >>> > > >>> > > >>> > Either way I think it's good to ship! :) > >>> > > >>> > > >>> > > >>> > Cheers > >>> > > >>> > > >>> > >> >
