And just for a bit of context: The failure rate is 6-7% under 500 changed lines, 19% for 500–2,000, and 56% over 2,000; 18% overall. A failure takes a median 3.4 h. So we should probably avoid doing this for large PRs for now - or make the PR smaller ;-).
On Tue, 10 Nov 2026 at 07:17, Hyukjin Kwon <[email protected]> wrote: > BTW, just for a bit of more context, I am slowly increasing the > parallelism. For now, I am monitoring the review results for all - it gives > me notification to my phone for every review :-). > > On Tue, 10 Nov 2026 at 06:47, Hyukjin Kwon <[email protected]> wrote: > >> Seems like this is working quite fine although sometimes it fails (and >> says to try again). Let me increase the parallelism to 8. So it reviews 8 >> at a time. >> >> On Wed, 4 Nov 2026 at 08:47, Holden Karau <[email protected]> wrote: >> >>> Very cool :) >>> >>> On Tue, Nov 3, 2026 at 2:51 PM Hyukjin Kwon via dev < >>> [email protected]> wrote: >>> >>>> Just FYI. Now, the bot is reviewing 4 at a time. The review takes >>>> around 1 hour. So sometimes it takes time. I am still monitoring. >>>> >>>> On Fri, 23 Oct 2026 at 07:09, Hyukjin Kwon <[email protected]> >>>> wrote: >>>> >>>>> FYI, I am monitoring and controlling details now. Effort, timeout, >>>>> security stuff. >>>>> >>>>> On Thu, 22 Oct 2026 at 22:51, Dongjoon Hyun <[email protected]> >>>>> wrote: >>>>> >>>>>> Got it. Thank you! >>>>>> >>>>>> Dongjoon. >>>>>> >>>>>> On 2026/10/22 11:44:04 Hyukjin Kwon wrote: >>>>>> > Just to reduce any surprise, I set another bot to monitor the bot >>>>>> itself. >>>>>> > If it does anything wrong, it halts immediately, and removes/reverts >>>>>> > whatever was wrong. >>>>>> > >>>>>> > For committers, please directly remove the `ai-review-requested` >>>>>> label if >>>>>> > there seems anything wrong. That would make the bot stop too. >>>>>> > >>>>>> > On Thu, 22 Oct 2026 at 20:21, Hyukjin Kwon <[email protected]> >>>>>> wrote: >>>>>> > >>>>>> > > Ok .. I managed to enable the bots. Please try >>>>>> `ai-review-requested` label >>>>>> > > out (only commiters can do). and please let me know if there are >>>>>> anything >>>>>> > > suspicious or not working well 👍 >>>>>> > > >>>>>> > > On Tue, 20 Oct 2026 at 08:31, Dongjoon Hyun <[email protected]> >>>>>> wrote: >>>>>> > > >>>>>> > >> Thank you so much for keeping working and sharing the status, >>>>>> Hyukjin. >>>>>> > >> >>>>>> > >> Dongjoon. >>>>>> > >> >>>>>> > >> On 2026/10/19 21:50:36 Hyukjin Kwon wrote: >>>>>> > >> > update: I am still struggling / testing. `ai-review-requested` >>>>>> label >>>>>> > >> does >>>>>> > >> > not work for now. >>>>>> > >> > >>>>>> > >> > On Fri, 16 Oct 2026 at 07:05, Hyukjin Kwon < >>>>>> [email protected]> >>>>>> > >> wrote: >>>>>> > >> > >>>>>> > >> > > headsup. I will stop the bot today. Will likely introduce >>>>>> > >> > > `ai-review-requested` label (only committers can set) to >>>>>> trigger the >>>>>> > >> review >>>>>> > >> > > most likely today for testing it out, see >>>>>> > >> > > https://github.com/apache/spark/pull/59502 >>>>>> > >> > > >>>>>> > >> > > On Mon, 12 Oct 2026 at 08:14, Hyukjin Kwon < >>>>>> [email protected]> >>>>>> > >> wrote: >>>>>> > >> > > >>>>>> > >> > >> Another quick heads up. >>>>>> > >> > >> >>>>>> > >> > >> - I am basically stopping (or rather not posting) the >>>>>> reviews when >>>>>> > >> > >> there are no findings. >>>>>> > >> > >> - If there are updates from the author, the review will >>>>>> be >>>>>> > >> triggered. >>>>>> > >> > >> To minimize the cost, I am not setting the effort as max >>>>>> but >>>>>> > >> rather trigger >>>>>> > >> > >> multiple times whenever the author pushes a newer commit. >>>>>> > >> > >> - I am looking especially through P3 review comments or >>>>>> not. >>>>>> > >> Please >>>>>> > >> > >> let me know if you guys are getting too many P3 nits, >>>>>> and if that >>>>>> > >> bugs you. >>>>>> > >> > >> >>>>>> > >> > >> >>>>>> > >> > >> >>>>>> > >> > >> On Thu, 8 Oct 2026 at 10:54, Hyukjin Kwon < >>>>>> [email protected]> >>>>>> > >> wrote: >>>>>> > >> > >> >>>>>> > >> > >>> Just quick headup, I am having some trouble with >>>>>> automation right >>>>>> > >> now. I >>>>>> > >> > >>> would need to test/run manually so it would be triggered >>>>>> like >>>>>> > >> everyday for >>>>>> > >> > >>> now. >>>>>> > >> > >>> >>>>>> > >> > >>> On Sat, 3 Oct 2026 at 01:53, Dongjoon Hyun < >>>>>> [email protected]> >>>>>> > >> wrote: >>>>>> > >> > >>> >>>>>> > >> > >>>> +1 >>>>>> > >> > >>>> >>>>>> > >> > >>>> Thanks, Hyukjin. >>>>>> > >> > >>>> >>>>>> > >> > >>>> Could we introduce a few safeguards for the new review >>>>>> bot? >>>>>> > >> > >>>> >>>>>> > >> > >>>> Specifically: >>>>>> > >> > >>>> - A GitHub label to opt out or disable the bot per PR. >>>>>> > >> > >>>> - Protections against excessive ASF Jira API calls. >>>>>> > >> > >>>> >>>>>> > >> > >>>> This would prevent the bot from being blocked/banned by >>>>>> ASF Jira >>>>>> > >> > >>>> and allow reviews to proceed even if Jira is down. It >>>>>> also helps >>>>>> > >> > >>>> manage worst-case scenarios where the bot leaves >>>>>> repetitive or >>>>>> > >> > >>>> unhelpful nitpicks. >>>>>> > >> > >>>> >>>>>> > >> > >>>> Best, >>>>>> > >> > >>>> Dongjoon >>>>>> > >> > >>>> >>>>>> > >> > >>>> On 2026-10-01 21:58 UTC Hyukjin Kwon wrote: >>>>>> > >> > >>>> > Hi all, >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > I've been running a trial of automated AI code review >>>>>> on new >>>>>> > >> > >>>> apache/spark >>>>>> > >> > >>>> > PRs, and I'd like to share how it works and get your >>>>>> feedback. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > For now the reviews are posted from my own GitHub >>>>>> account >>>>>> > >> > >>>> (@HyukjinKwon) >>>>>> > >> > >>>> > while I check that it behaves well. Once it's proven, I >>>>>> plan to >>>>>> > >> move >>>>>> > >> > >>>> it to >>>>>> > >> > >>>> > the https://github.com/spark-test account, so the >>>>>> reviews are >>>>>> > >> clearly >>>>>> > >> > >>>> > separate from my personal ones. I plan to run this for >>>>>> the next >>>>>> > >> week. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *What it does* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > Each review is a full, multi-pass review of the PR >>>>>> diff, run by >>>>>> > >> Claude >>>>>> > >> > >>>> > (via Claude Code). Findings are checked against the >>>>>> code before >>>>>> > >> > >>>> they're >>>>>> > >> > >>>> > posted. They're posted as a normal GitHub review: a >>>>>> short >>>>>> > >> summary plus >>>>>> > >> > >>>> > inline comments, each labeled with a severity, e.g. >>>>>> "Non-blocking >>>>>> > >> > >>>> (P2)" >>>>>> > >> > >>>> > or "Nit (P3)". Example: >>>>>> > >> > >>>> > >>>>>> > >> > >>>> >>>>>> > >> >>>>>> https://github.com/apache/spark/pull/59184#pullrequestreview-5379759787 >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *No LGTM, no approvals* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > - It never approves a PR and never posts "LGTM". Every >>>>>> review is >>>>>> > >> a >>>>>> > >> > >>>> plain >>>>>> > >> > >>>> > "Comment" review, so it doesn't count as an approval >>>>>> or a >>>>>> > >> change >>>>>> > >> > >>>> > request, and it has no effect on merging. >>>>>> > >> > >>>> > - If a review finds nothing new, it posts nothing at >>>>>> all: no "no >>>>>> > >> > >>>> > findings" or "all comments addressed" message. >>>>>> > >> > >>>> > - It doesn't push commits or edit anyone's PR. >>>>>> > >> > >>>> > - Human review and committer judgement are unchanged. >>>>>> Please >>>>>> > >> treat its >>>>>> > >> > >>>> > comments like any other reviewer's: feel free to push >>>>>> back or >>>>>> > >> > >>>> ignore. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *When a review is triggered* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > - A new PR is reviewed shortly after it's opened. Draft >>>>>> PRs are >>>>>> > >> > >>>> reviewed >>>>>> > >> > >>>> > once they're marked ready for review. >>>>>> > >> > >>>> > - After a review posts findings, the bot waits. It >>>>>> reviews again >>>>>> > >> only >>>>>> > >> > >>>> > when the author pushes new commits. >>>>>> > >> > >>>> > - It also waits while a PR has open review comments >>>>>> from anyone >>>>>> > >> that >>>>>> > >> > >>>> are >>>>>> > >> > >>>> > newer than the latest commit, so it doesn't pile on >>>>>> while the >>>>>> > >> > >>>> author is >>>>>> > >> > >>>> > still responding to reviewers. >>>>>> > >> > >>>> > - Replying in a thread without pushing doesn't trigger a >>>>>> > >> re-review. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *When it stops* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > - When a review finds nothing new. The PR gets a fresh >>>>>> review if >>>>>> > >> the >>>>>> > >> > >>>> > author pushes again later. >>>>>> > >> > >>>> > - When the PR is merged or closed. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *Early numbers* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > Over the first ~12 hours (Oct 1), it covered 10 PRs, >>>>>> including 5 >>>>>> > >> > >>>> > re-reviews after the author pushed. It posted 7 reviews >>>>>> with 14 >>>>>> > >> > >>>> findings >>>>>> > >> > >>>> > in total, mostly P3 nits. Four PRs came back clean and >>>>>> got no >>>>>> > >> comment. >>>>>> > >> > >>>> > Several authors pushed fixes, and the follow-up reviews >>>>>> found >>>>>> > >> nothing >>>>>> > >> > >>>> > new. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > *Feedback* >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > Please let me know if the comments are useful, noisy, >>>>>> or wrong, >>>>>> > >> either >>>>>> > >> > >>>> > here or on the PR. If you'd rather it not review your >>>>>> PRs, tell >>>>>> > >> me and >>>>>> > >> > >>>> > I'll exclude them. I'm also happy to adjust when it >>>>>> triggers >>>>>> > >> based on >>>>>> > >> > >>>> > what the community prefers. >>>>>> > >> > >>>> > >>>>>> > >> > >>>> > Thanks, >>>>>> > >> > >>>> > Hyukjin >>>>>> > >> > >>>> > >>>>>> > >> > >>>> >>>>>> > >> > >>>> >>>>>> > >> >>>>>> --------------------------------------------------------------------- >>>>>> > >> > >>>> To unsubscribe e-mail: [email protected] >>>>>> > >> > >>>> >>>>>> > >> > >>>> >>>>>> > >> > >>>>>> > >> >>>>>> > >> >>>>>> --------------------------------------------------------------------- >>>>>> > >> To unsubscribe e-mail: [email protected] >>>>>> > >> >>>>>> > >> >>>>>> > >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe e-mail: [email protected] >>>>>> >>>>>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>
