Hi Yufei, I'm open to trying Orca for PR reviews with the reservation that its quality will be subject to "review" (by the Polaris community) as we experience it in real-life situations.
However, I do not think it will reduce the load on human reviewers. LLMs can help greatly with identifying logical errors and hidden mistakes. However, I'm sure many people already use local agents for this purpose. Still, at this time, LLMs cannot guide the PR author on the overall Polaris design and, more importantly, on the long term evolution that the community is discussing / envisioning. Human input will still be critical because of the latter, IMHO. Cheers, Dmitri. On Tue, Oct 20, 2026 at 6:28 PM Yufei Gu <[email protected]> wrote: > Hi folks, > > As PR volume grows, review is becoming a bottleneck. I’d like to propose > using Orca, an AI review bot, to help identify potential issues in Polaris > PRs and reduce the load on human reviewers. Other communities(e.g., > Spark[1], Airflow[2]) are working on a similar idea. > > For Polaris, the initial Orca review results look more useful than Copilot. > Orca uses additional project context from a Polaris LLM wiki covering > architecture, conventions, and review guidance to help produce more > relevant feedback. Check Polaris PR #5532 > <https://github.com/apache/polaris/pull/5532> to see how it works. > > A few principles for the proposed workflow: > > - *Clear attribution:* Every review would explicitly state that it is > AI-generated. > - *No approvals or LGTMs:* The bot would provide comments only. It would > not approve PRs or make merge decisions. > - *A separate GitHub account:* Reviews would come from orca-review-bot, > clearly distinguishable from human reviews. > > Reviews are triggered manually today. As the next step, I'm working on > automating execution so that someone can explicitly request a review from > Orca on a PR. This would keep reviews opt-in while removing the manual work > needed to run them. > > There are two limitations to keep in mind: > > - *False positives:* Like a human reviewer, the bot can misunderstand > code or flag issues that aren't problems. Please push back on incorrect > findings. > - *False negatives:* The bot will miss issues, just as human reviewers > do. An absence of findings does not mean a PR is ready to merge. > Committers > remain responsible for the final review and merge decision. > > I'm starting an opt-in trial today. Feel free to share any feedback on > review quality, the triggering workflow, and ways to keep comment noise > manageable; this feedback would be especially helpful. > 1. https://www.mail-archive.com/dev%40spark.apache.org/msg36376.html > 2. https://www.mail-archive.com/[email protected]/msg521119.html > > Thanks! > Yufei >
