Hi all, Thanks Yufei for driving this, and JB for raising the legal points. I have no objections to a trial, but we need to settle a few things first.
I couldn't tell which Orca this is or who pays for it. The only matching tool I found is OrcaCode Review (MIT, built on Open Code Review, which is Apache-2.0). It routes through OrcaRouter, needs an OrcaRouter API key, and bills inference at the upstream providers' rates. I found no free tier for open-source or ASF projects, only OrcaRouter's general free tier. If that is the tool, it needs a funded API key. Could you share which tool and model(s) it is, whose account and API key the bot uses, who pays for inference and how that cost is bounded, what data it retains or trains on, and which GitHub permissions it needs? Also, where does the Polaris LLM wiki live, and is it public? Without those answers, JB's suggestion to check and share the model/provider and its terms of use (per the ASF generative tooling guidance) can't be followed. Depending on the answers, a project-funded or personally funded key may be a governance question for the PMC, not just a legal one. Copilot code review is already enabled in .asf.yaml (non-draft PRs, no review on push). Its instructions in .github/copilot-instructions.md [1] are deliberately conservative: stay silent unless there's a concrete sign that docs, security, or tests are missing. How would Orca coexist with it? Two bots on the same diff will produce duplicates and noise. It might also be worth checking whether using Copilot more extensively, with broader instructions, already covers much of what we want. Running both on a handful of recent PRs and comparing findings would help before deciding. The same terms and provider questions apply to Copilot, so I'd cover both in the CONTRIBUTING.md update JB suggested. The proposal only considers Orca, but there are alternatives worth a look: - Greptile: free for qualifying open-source projects (public, OSI-licensed, at least 50 stars), so no own AI subscription is needed. Note that it may train on de-identified data unless an org admin opts out. - CodeRabbit: free for public repos, and states it doesn't train on customer code. - Qodo PR-Agent: open source, run as a self-hosted action with our own LLM keys, so like Orca it costs token spend. - A committer-run model like Spark's trial (Claude via Claude Code) or Airflow's maintainer review skill, which uses the committer's own subscription. A short comparison on who pays, what data leaves the project, and whether a third-party app is installed would make the trade-offs concrete. This is from my own reading of public vendor pages, so please correct anything wrong. If this ends up as a GitHub app rather than just a bot account, AFAIK ASF INFRA would need to approve it on apache/polaris. I'd also exclude security-sensitive PRs from bot review. +1 to JB's point about labeling and attribution: label every inline comment, not just the summary, and keep the "comments only, no approvals" rule. Robert [1] https://github.com/apache/polaris/blob/b2a339e484cf75a577f5caaa17dc52e690c128a8/.github/copilot-instructions.md On Wed, Oct 21, 2026 at 6:55 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi guys, > > Thanks Yufei for starting this discussion. > > I'm +1 to try Orca as an opt-in trial (it's funny, Orca is also the name of > devices I'm using https://getorca.com/ :)). > I think we all agree it's a great help for catching mistakes, but the > design and final merge decision remain subject to human review. > > I would like to remind us to stay aligned with the ASF Generative Tooling > guidance (https://www.apache.org/legal/generative-tooling.html). Two parts > of it apply here: > - The guidance says that AI-generated text published publicly must either > be reveiwed before publication by a person who take reponsability for it, > or clearly state it was generated by AI (EU AI Act, in force since August > 2026). A bot posting review comments directly on public PRs fails the > second case. Your "clear attribution" prniciple covers this, and on PR > #5332 both the review summary and each inline comment already carry the > label. Let's keep it that way on every comment, as inline comments are > often read on their own (notifications, emails). > - If Orca proposes code (for instance GitHub suggestion blocks) and the PR > author applies it, that code becomes an AI-generated contriibution. The > guidance then asks the contributor to ensure that the tool's terms of use > don't restrict the output in a way inconsistent with the Open Source > Definition and grant enough rights to contribute it under the Apache > License 2.0. Also the output must not include third-party material (or only > material used under a compatible license). I think we should check and > share which model/provider Orca is using, with a pointer to its terms of > use. With that contributors can check without guessing. The guidance also > recommends a "Generated-by: <tool> <version>" (or "Co-authored-by:") in > commit message, which fits the disclosure we already encourage in our > CONTRIBUTING.md ( > > https://github.com/apache/polaris/blob/main/CONTRIBUTING.md#guidelines-for-ai-assisted-contributions > ). > > Once the trial gives us some feedback, I think we should update > CONTRIBUTING.md with model/provider with the tool used by contributors. > > Regards > JB > > On Wed, Oct 21, 2026 at 12:28 AM Yufei Gu <[email protected]> wrote: > > > Hi folks, > > > > As PR volume grows, review is becoming a bottleneck. I’d like to propose > > using Orca, an AI review bot, to help identify potential issues in > Polaris > > PRs and reduce the load on human reviewers. Other communities(e.g., > > Spark[1], Airflow[2]) are working on a similar idea. > > > > For Polaris, the initial Orca review results look more useful than > Copilot. > > Orca uses additional project context from a Polaris LLM wiki covering > > architecture, conventions, and review guidance to help produce more > > relevant feedback. Check Polaris PR #5532 > > <https://github.com/apache/polaris/pull/5532> to see how it works. > > > > A few principles for the proposed workflow: > > > > - *Clear attribution:* Every review would explicitly state that it is > > AI-generated. > > - *No approvals or LGTMs:* The bot would provide comments only. It > would > > not approve PRs or make merge decisions. > > - *A separate GitHub account:* Reviews would come from > orca-review-bot, > > clearly distinguishable from human reviews. > > > > Reviews are triggered manually today. As the next step, I'm working on > > automating execution so that someone can explicitly request a review from > > Orca on a PR. This would keep reviews opt-in while removing the manual > work > > needed to run them. > > > > There are two limitations to keep in mind: > > > > - *False positives:* Like a human reviewer, the bot can misunderstand > > code or flag issues that aren't problems. Please push back on > incorrect > > findings. > > - *False negatives:* The bot will miss issues, just as human reviewers > > do. An absence of findings does not mean a PR is ready to merge. > > Committers > > remain responsible for the final review and merge decision. > > > > I'm starting an opt-in trial today. Feel free to share any feedback on > > review quality, the triggering workflow, and ways to keep comment noise > > manageable; this feedback would be especially helpful. > > 1. https://www.mail-archive.com/dev%40spark.apache.org/msg36376.html > > 2. > https://www.mail-archive.com/[email protected]/msg521119.html > > > > Thanks! > > Yufei > > >
