Hi JB, Robert, and Vignesh, Thanks for the feedback!
JB, agreed. We should label every comment as AI generated, including inline comments, and share the model/provider and terms for anyone using its code suggestions. Both are already included in the initial version of the Orca review bot. Robert, fair questions. I should have explained the setup in the original email. Orca is a project I’ve been working on for a while. It can use different LLMs, such as models from Anthropic or OpenAI. My employer currently covers the cost. We’ve also discussed whether ASF Infra could sponsor it, and I’ll follow up on that. The LLM wiki is another project I’ve been working on. I plan to open source both projects since they could be useful beyond Polaris. I’ll share more details, including where to find the wiki, in a follow up. I’m open to other tools too. Orca has looked useful in the reviews I’ve tried, but comparing it with Copilot on a few PRs makes sense. Vignesh, thanks for checking #5717. Your point about missing context is a good one. The current LLM wiki is out of date, which may have contributed to the mismatch in that review. I’m still working on the pipeline to keep it current. Getting the bot to retrieve the right context is another challenge. Even with a complete wiki, it may miss relevant information because the whole wiki could be too large to fit in the model’s context window. I also agree that authors shouldn’t feel they have to address every bot comment. Please push back when it gets something wrong. Anything that changes the design or expands the PR scope should go through a human reviewer first. Let’s keep the trial small and see whether it actually saves review time. Yufei On Wed, Oct 21, 2026 at 6:18 AM vignesh a <[email protected]> wrote: > Hi all, > > Thanks Yufei for starting the trial, and Dmitri, JB and Robert for the > points so far. Sharing some feedback from real usage. > > I went through the Orca review on PR #5717 [1]. A few comments were > valid on their own, like the Kafka mapper duplication, "type" becoming > the contract, and the "data.source" vs CE "source" naming. > > What it missed was the context around the PR: the dev list discussion, > the sequencing with #5084 [2], and that this PR is meant to be > mapper-only. In Polaris a lot of design direction is agreed on the > list and in syncs, and the bot can't see that yet. > > > > *My concern is newer or part-time contributors treating bot comments > asmust-do and growing the PR scope. Reviewers would then be reviewingboth > the diff and the bot thread.* > > A suggestion for the trial: treat Orca comments like any other nit. > The author can reply, but only changes code once a human reviewer > agrees the finding is valid. It may also help to link related dev > threads or issues in the PR description, so the bot (and humans) get > that context.. > > Robert's idea of comparing it with Copilot on a few recent PRs sounds > like a good way to judge it. Happy to share more as I see it on other > PRs. > > Thanks, > Vignesh > > [1] https://github.com/apache/polaris/pull/5717 > [2] https://github.com/apache/polaris/pull/5084 >
