Thanks for working on this, Hyukjin. The reviews are often helpful and improve the quality of contributions.
Two questions: 1. Is it possible to summon the bot using the usual reviewer UI on GitHub vs. needing to add a label? That would be a bit cleaner. 2. Can we configure the bot to be more brief? Many of its review comments are extremely long, and this encourages the human receiving the feedback to not read it and instead ask their own bot to summarize and/or respond. So then we have two bots talking to one another via a human proxy. 😄 I know that sometimes the added verbiage is useful as context or proof of work. Could we have the bot structure each comment as a brief, executive summary and tuck any additional detail away behind a collapsed `<details>` element? GitHub supports those. Nick > On Nov 9, 2026, at 5:24 PM, Hyukjin Kwon via dev <[email protected]> wrote: > > And just for a bit of context: > > The failure rate is 6-7% under 500 changed lines, 19% for 500–2,000, and 56% > over 2,000; 18% overall. A failure takes a median 3.4 h. > So we should probably avoid doing this for large PRs for now - or make the PR > smaller ;-). > > > On Tue, 10 Nov 2026 at 07:17, Hyukjin Kwon <[email protected] > <mailto:[email protected]>> wrote: >> BTW, just for a bit of more context, I am slowly increasing the parallelism. >> For now, I am monitoring the review results for all - it gives me >> notification to my phone for every review :-). >> >> On Tue, 10 Nov 2026 at 06:47, Hyukjin Kwon <[email protected] >> <mailto:[email protected]>> wrote: >>> Seems like this is working quite fine although sometimes it fails (and says >>> to try again). Let me increase the parallelism to 8. So it reviews 8 at a >>> time. >>> >>> On Wed, 4 Nov 2026 at 08:47, Holden Karau <[email protected] >>> <mailto:[email protected]>> wrote: >>>> Very cool :) >>>> >>>> On Tue, Nov 3, 2026 at 2:51 PM Hyukjin Kwon via dev <[email protected] >>>> <mailto:[email protected]>> wrote: >>>>> Just FYI. Now, the bot is reviewing 4 at a time. The review takes around >>>>> 1 hour. So sometimes it takes time. I am still monitoring. >>>>> >>>>> On Fri, 23 Oct 2026 at 07:09, Hyukjin Kwon <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>>> FYI, I am monitoring and controlling details now. Effort, timeout, >>>>>> security stuff. >>>>>> >>>>>> On Thu, 22 Oct 2026 at 22:51, Dongjoon Hyun <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>>> Got it. Thank you! >>>>>>> >>>>>>> Dongjoon. >>>>>>> >>>>>>> On 2026/10/22 11:44:04 Hyukjin Kwon wrote: >>>>>>> > Just to reduce any surprise, I set another bot to monitor the bot >>>>>>> > itself. >>>>>>> > If it does anything wrong, it halts immediately, and removes/reverts >>>>>>> > whatever was wrong. >>>>>>> > >>>>>>> > For committers, please directly remove the `ai-review-requested` >>>>>>> > label if >>>>>>> > there seems anything wrong. That would make the bot stop too. >>>>>>> > >>>>>>> > On Thu, 22 Oct 2026 at 20:21, Hyukjin Kwon <[email protected] >>>>>>> > <mailto:[email protected]>> wrote: >>>>>>> > >>>>>>> > > Ok .. I managed to enable the bots. Please try >>>>>>> > > `ai-review-requested` label >>>>>>> > > out (only commiters can do). and please let me know if there are >>>>>>> > > anything >>>>>>> > > suspicious or not working well 👍 >>>>>>> > > >>>>>>> > > On Tue, 20 Oct 2026 at 08:31, Dongjoon Hyun <[email protected] >>>>>>> > > <mailto:[email protected]>> wrote: >>>>>>> > > >>>>>>> > >> Thank you so much for keeping working and sharing the status, >>>>>>> > >> Hyukjin. >>>>>>> > >> >>>>>>> > >> Dongjoon. >>>>>>> > >> >>>>>>> > >> On 2026/10/19 21:50:36 Hyukjin Kwon wrote: >>>>>>> > >> > update: I am still struggling / testing. `ai-review-requested` >>>>>>> > >> > label >>>>>>> > >> does >>>>>>> > >> > not work for now. >>>>>>> > >> > >>>>>>> > >> > On Fri, 16 Oct 2026 at 07:05, Hyukjin Kwon <[email protected] >>>>>>> > >> > <mailto:[email protected]>> >>>>>>> > >> wrote: >>>>>>> > >> > >>>>>>> > >> > > headsup. I will stop the bot today. Will likely introduce >>>>>>> > >> > > `ai-review-requested` label (only committers can set) to >>>>>>> > >> > > trigger the >>>>>>> > >> review >>>>>>> > >> > > most likely today for testing it out, see >>>>>>> > >> > > https://github.com/apache/spark/pull/59502 >>>>>>> > >> > > >>>>>>> > >> > > On Mon, 12 Oct 2026 at 08:14, Hyukjin Kwon >>>>>>> > >> > > <[email protected] <mailto:[email protected]>> >>>>>>> > >> wrote: >>>>>>> > >> > > >>>>>>> > >> > >> Another quick heads up. >>>>>>> > >> > >> >>>>>>> > >> > >> - I am basically stopping (or rather not posting) the >>>>>>> > >> > >> reviews when >>>>>>> > >> > >> there are no findings. >>>>>>> > >> > >> - If there are updates from the author, the review will be >>>>>>> > >> triggered. >>>>>>> > >> > >> To minimize the cost, I am not setting the effort as max >>>>>>> > >> > >> but >>>>>>> > >> rather trigger >>>>>>> > >> > >> multiple times whenever the author pushes a newer commit. >>>>>>> > >> > >> - I am looking especially through P3 review comments or >>>>>>> > >> > >> not. >>>>>>> > >> Please >>>>>>> > >> > >> let me know if you guys are getting too many P3 nits, and >>>>>>> > >> > >> if that >>>>>>> > >> bugs you. >>>>>>> > >> > >> >>>>>>> > >> > >> >>>>>>> > >> > >> >>>>>>> > >> > >> On Thu, 8 Oct 2026 at 10:54, Hyukjin Kwon >>>>>>> > >> > >> <[email protected] <mailto:[email protected]>> >>>>>>> > >> wrote: >>>>>>> > >> > >> >>>>>>> > >> > >>> Just quick headup, I am having some trouble with automation >>>>>>> > >> > >>> right >>>>>>> > >> now. I >>>>>>> > >> > >>> would need to test/run manually so it would be triggered like >>>>>>> > >> everyday for >>>>>>> > >> > >>> now. >>>>>>> > >> > >>> >>>>>>> > >> > >>> On Sat, 3 Oct 2026 at 01:53, Dongjoon Hyun >>>>>>> > >> > >>> <[email protected] <mailto:[email protected]>> >>>>>>> > >> wrote: >>>>>>> > >> > >>> >>>>>>> > >> > >>>> +1 >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> Thanks, Hyukjin. >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> Could we introduce a few safeguards for the new review bot? >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> Specifically: >>>>>>> > >> > >>>> - A GitHub label to opt out or disable the bot per PR. >>>>>>> > >> > >>>> - Protections against excessive ASF Jira API calls. >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> This would prevent the bot from being blocked/banned by ASF >>>>>>> > >> > >>>> Jira >>>>>>> > >> > >>>> and allow reviews to proceed even if Jira is down. It also >>>>>>> > >> > >>>> helps >>>>>>> > >> > >>>> manage worst-case scenarios where the bot leaves repetitive >>>>>>> > >> > >>>> or >>>>>>> > >> > >>>> unhelpful nitpicks. >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> Best, >>>>>>> > >> > >>>> Dongjoon >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> On 2026-10-01 21:58 UTC Hyukjin Kwon wrote: >>>>>>> > >> > >>>> > Hi all, >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > I've been running a trial of automated AI code review on >>>>>>> > >> > >>>> > new >>>>>>> > >> > >>>> apache/spark >>>>>>> > >> > >>>> > PRs, and I'd like to share how it works and get your >>>>>>> > >> > >>>> > feedback. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > For now the reviews are posted from my own GitHub account >>>>>>> > >> > >>>> (@HyukjinKwon) >>>>>>> > >> > >>>> > while I check that it behaves well. Once it's proven, I >>>>>>> > >> > >>>> > plan to >>>>>>> > >> move >>>>>>> > >> > >>>> it to >>>>>>> > >> > >>>> > the https://github.com/spark-test account, so the reviews >>>>>>> > >> > >>>> > are >>>>>>> > >> clearly >>>>>>> > >> > >>>> > separate from my personal ones. I plan to run this for >>>>>>> > >> > >>>> > the next >>>>>>> > >> week. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *What it does* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > Each review is a full, multi-pass review of the PR diff, >>>>>>> > >> > >>>> > run by >>>>>>> > >> Claude >>>>>>> > >> > >>>> > (via Claude Code). Findings are checked against the code >>>>>>> > >> > >>>> > before >>>>>>> > >> > >>>> they're >>>>>>> > >> > >>>> > posted. They're posted as a normal GitHub review: a short >>>>>>> > >> summary plus >>>>>>> > >> > >>>> > inline comments, each labeled with a severity, e.g. >>>>>>> > >> > >>>> > "Non-blocking >>>>>>> > >> > >>>> (P2)" >>>>>>> > >> > >>>> > or "Nit (P3)". Example: >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> >>>>>>> > >> https://github.com/apache/spark/pull/59184#pullrequestreview-5379759787 >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *No LGTM, no approvals* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > - It never approves a PR and never posts "LGTM". Every >>>>>>> > >> > >>>> > review is >>>>>>> > >> a >>>>>>> > >> > >>>> plain >>>>>>> > >> > >>>> > "Comment" review, so it doesn't count as an approval or >>>>>>> > >> > >>>> > a >>>>>>> > >> change >>>>>>> > >> > >>>> > request, and it has no effect on merging. >>>>>>> > >> > >>>> > - If a review finds nothing new, it posts nothing at all: >>>>>>> > >> > >>>> > no "no >>>>>>> > >> > >>>> > findings" or "all comments addressed" message. >>>>>>> > >> > >>>> > - It doesn't push commits or edit anyone's PR. >>>>>>> > >> > >>>> > - Human review and committer judgement are unchanged. >>>>>>> > >> > >>>> > Please >>>>>>> > >> treat its >>>>>>> > >> > >>>> > comments like any other reviewer's: feel free to push >>>>>>> > >> > >>>> > back or >>>>>>> > >> > >>>> ignore. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *When a review is triggered* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > - A new PR is reviewed shortly after it's opened. Draft >>>>>>> > >> > >>>> > PRs are >>>>>>> > >> > >>>> reviewed >>>>>>> > >> > >>>> > once they're marked ready for review. >>>>>>> > >> > >>>> > - After a review posts findings, the bot waits. It >>>>>>> > >> > >>>> > reviews again >>>>>>> > >> only >>>>>>> > >> > >>>> > when the author pushes new commits. >>>>>>> > >> > >>>> > - It also waits while a PR has open review comments from >>>>>>> > >> > >>>> > anyone >>>>>>> > >> that >>>>>>> > >> > >>>> are >>>>>>> > >> > >>>> > newer than the latest commit, so it doesn't pile on >>>>>>> > >> > >>>> > while the >>>>>>> > >> > >>>> author is >>>>>>> > >> > >>>> > still responding to reviewers. >>>>>>> > >> > >>>> > - Replying in a thread without pushing doesn't trigger a >>>>>>> > >> re-review. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *When it stops* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > - When a review finds nothing new. The PR gets a fresh >>>>>>> > >> > >>>> > review if >>>>>>> > >> the >>>>>>> > >> > >>>> > author pushes again later. >>>>>>> > >> > >>>> > - When the PR is merged or closed. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *Early numbers* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > Over the first ~12 hours (Oct 1), it covered 10 PRs, >>>>>>> > >> > >>>> > including 5 >>>>>>> > >> > >>>> > re-reviews after the author pushed. It posted 7 reviews >>>>>>> > >> > >>>> > with 14 >>>>>>> > >> > >>>> findings >>>>>>> > >> > >>>> > in total, mostly P3 nits. Four PRs came back clean and >>>>>>> > >> > >>>> > got no >>>>>>> > >> comment. >>>>>>> > >> > >>>> > Several authors pushed fixes, and the follow-up reviews >>>>>>> > >> > >>>> > found >>>>>>> > >> nothing >>>>>>> > >> > >>>> > new. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > *Feedback* >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > Please let me know if the comments are useful, noisy, or >>>>>>> > >> > >>>> > wrong, >>>>>>> > >> either >>>>>>> > >> > >>>> > here or on the PR. If you'd rather it not review your >>>>>>> > >> > >>>> > PRs, tell >>>>>>> > >> me and >>>>>>> > >> > >>>> > I'll exclude them. I'm also happy to adjust when it >>>>>>> > >> > >>>> > triggers >>>>>>> > >> based on >>>>>>> > >> > >>>> > what the community prefers. >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> > Thanks, >>>>>>> > >> > >>>> > Hyukjin >>>>>>> > >> > >>>> > >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> >>>>>>> > >> --------------------------------------------------------------------- >>>>>>> > >> > >>>> To unsubscribe e-mail: [email protected] >>>>>>> > >> > >>>> <mailto:[email protected]> >>>>>>> > >> > >>>> >>>>>>> > >> > >>>> >>>>>>> > >> > >>>>>>> > >> >>>>>>> > >> --------------------------------------------------------------------- >>>>>>> > >> To unsubscribe e-mail: [email protected] >>>>>>> > >> <mailto:[email protected]> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: [email protected] >>>>>>> <mailto:[email protected]> >>>>>>> >>>> >>>> >>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> Pronouns: she/her
