BTW, I see this AI review fails for large PRs mostly for timeout (3 hours by default). I will increase timeout some more.
On Fri, 13 Nov 2026 at 06:58, Hyukjin Kwon <[email protected]> wrote: > Ah for 1. you mean to request review to spark-test account at "Reviewers" > UI? I think that's a good idea. Let me take a look. > > 2. .. :-). I wonder if longer details are better or short ones are better. > The main problem about this is security stuff .. this contains some > credentials when it runs in fact, and I would like to avoid others to > manipulate it for now. > But I do agree that we would need some ways to configure this though. I > think this one I would have to think about more ;-(. > > > On Fri, 13 Nov 2026 at 00:39, Nicholas Chammas <[email protected]> > wrote: > >> Thanks for working on this, Hyukjin. The reviews are often helpful and >> improve the quality of contributions. >> >> Two questions: >> >> 1. Is it possible to summon the bot using the usual reviewer UI on GitHub >> vs. needing to add a label? That would be a bit cleaner. >> >> 2. Can we configure the bot to be more brief? Many of its review comments >> are extremely long, and this encourages the human receiving the feedback to >> not read it and instead ask their own bot to summarize and/or respond. So >> then we have two bots talking to one another via a human proxy. 😄 >> >> I know that sometimes the added verbiage is useful as context or proof of >> work. Could we have the bot structure each comment as a brief, executive >> summary and tuck any additional detail away behind a collapsed `<details>` >> element? GitHub supports those. >> >> Nick >> >> >> On Nov 9, 2026, at 5:24 PM, Hyukjin Kwon via dev <[email protected]> >> wrote: >> >> And just for a bit of context: >> >> The failure rate is 6-7% under 500 changed lines, 19% for 500–2,000, and >> 56% over 2,000; 18% overall. A failure takes a median 3.4 h. >> So we should probably avoid doing this for large PRs for now - or make >> the PR smaller ;-). >> >> >> On Tue, 10 Nov 2026 at 07:17, Hyukjin Kwon <[email protected]> wrote: >> >>> BTW, just for a bit of more context, I am slowly increasing the >>> parallelism. For now, I am monitoring the review results for all - it gives >>> me notification to my phone for every review :-). >>> >>> On Tue, 10 Nov 2026 at 06:47, Hyukjin Kwon <[email protected]> wrote: >>> >>>> Seems like this is working quite fine although sometimes it fails (and >>>> says to try again). Let me increase the parallelism to 8. So it reviews 8 >>>> at a time. >>>> >>>> On Wed, 4 Nov 2026 at 08:47, Holden Karau <[email protected]> >>>> wrote: >>>> >>>>> Very cool :) >>>>> >>>>> On Tue, Nov 3, 2026 at 2:51 PM Hyukjin Kwon via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> Just FYI. Now, the bot is reviewing 4 at a time. The review takes >>>>>> around 1 hour. So sometimes it takes time. I am still monitoring. >>>>>> >>>>>> On Fri, 23 Oct 2026 at 07:09, Hyukjin Kwon <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> FYI, I am monitoring and controlling details now. Effort, timeout, >>>>>>> security stuff. >>>>>>> >>>>>>> On Thu, 22 Oct 2026 at 22:51, Dongjoon Hyun <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Got it. Thank you! >>>>>>>> >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>> On 2026/10/22 11:44:04 Hyukjin Kwon wrote: >>>>>>>> > Just to reduce any surprise, I set another bot to monitor the bot >>>>>>>> itself. >>>>>>>> > If it does anything wrong, it halts immediately, and >>>>>>>> removes/reverts >>>>>>>> > whatever was wrong. >>>>>>>> > >>>>>>>> > For committers, please directly remove the `ai-review-requested` >>>>>>>> label if >>>>>>>> > there seems anything wrong. That would make the bot stop too. >>>>>>>> > >>>>>>>> > On Thu, 22 Oct 2026 at 20:21, Hyukjin Kwon <[email protected]> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > > Ok .. I managed to enable the bots. Please try >>>>>>>> `ai-review-requested` label >>>>>>>> > > out (only commiters can do). and please let me know if there >>>>>>>> are anything >>>>>>>> > > suspicious or not working well 👍 >>>>>>>> > > >>>>>>>> > > On Tue, 20 Oct 2026 at 08:31, Dongjoon Hyun < >>>>>>>> [email protected]> wrote: >>>>>>>> > > >>>>>>>> > >> Thank you so much for keeping working and sharing the status, >>>>>>>> Hyukjin. >>>>>>>> > >> >>>>>>>> > >> Dongjoon. >>>>>>>> > >> >>>>>>>> > >> On 2026/10/19 21:50:36 Hyukjin Kwon wrote: >>>>>>>> > >> > update: I am still struggling / testing. >>>>>>>> `ai-review-requested` label >>>>>>>> > >> does >>>>>>>> > >> > not work for now. >>>>>>>> > >> > >>>>>>>> > >> > On Fri, 16 Oct 2026 at 07:05, Hyukjin Kwon < >>>>>>>> [email protected]> >>>>>>>> > >> wrote: >>>>>>>> > >> > >>>>>>>> > >> > > headsup. I will stop the bot today. Will likely introduce >>>>>>>> > >> > > `ai-review-requested` label (only committers can set) to >>>>>>>> trigger the >>>>>>>> > >> review >>>>>>>> > >> > > most likely today for testing it out, see >>>>>>>> > >> > > https://github.com/apache/spark/pull/59502 >>>>>>>> > >> > > >>>>>>>> > >> > > On Mon, 12 Oct 2026 at 08:14, Hyukjin Kwon < >>>>>>>> [email protected]> >>>>>>>> > >> wrote: >>>>>>>> > >> > > >>>>>>>> > >> > >> Another quick heads up. >>>>>>>> > >> > >> >>>>>>>> > >> > >> - I am basically stopping (or rather not posting) the >>>>>>>> reviews when >>>>>>>> > >> > >> there are no findings. >>>>>>>> > >> > >> - If there are updates from the author, the review >>>>>>>> will be >>>>>>>> > >> triggered. >>>>>>>> > >> > >> To minimize the cost, I am not setting the effort as >>>>>>>> max but >>>>>>>> > >> rather trigger >>>>>>>> > >> > >> multiple times whenever the author pushes a newer >>>>>>>> commit. >>>>>>>> > >> > >> - I am looking especially through P3 review comments >>>>>>>> or not. >>>>>>>> > >> Please >>>>>>>> > >> > >> let me know if you guys are getting too many P3 nits, >>>>>>>> and if that >>>>>>>> > >> bugs you. >>>>>>>> > >> > >> >>>>>>>> > >> > >> >>>>>>>> > >> > >> >>>>>>>> > >> > >> On Thu, 8 Oct 2026 at 10:54, Hyukjin Kwon < >>>>>>>> [email protected]> >>>>>>>> > >> wrote: >>>>>>>> > >> > >> >>>>>>>> > >> > >>> Just quick headup, I am having some trouble with >>>>>>>> automation right >>>>>>>> > >> now. I >>>>>>>> > >> > >>> would need to test/run manually so it would be triggered >>>>>>>> like >>>>>>>> > >> everyday for >>>>>>>> > >> > >>> now. >>>>>>>> > >> > >>> >>>>>>>> > >> > >>> On Sat, 3 Oct 2026 at 01:53, Dongjoon Hyun < >>>>>>>> [email protected]> >>>>>>>> > >> wrote: >>>>>>>> > >> > >>> >>>>>>>> > >> > >>>> +1 >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> Thanks, Hyukjin. >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> Could we introduce a few safeguards for the new review >>>>>>>> bot? >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> Specifically: >>>>>>>> > >> > >>>> - A GitHub label to opt out or disable the bot per PR. >>>>>>>> > >> > >>>> - Protections against excessive ASF Jira API calls. >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> This would prevent the bot from being blocked/banned by >>>>>>>> ASF Jira >>>>>>>> > >> > >>>> and allow reviews to proceed even if Jira is down. It >>>>>>>> also helps >>>>>>>> > >> > >>>> manage worst-case scenarios where the bot leaves >>>>>>>> repetitive or >>>>>>>> > >> > >>>> unhelpful nitpicks. >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> Best, >>>>>>>> > >> > >>>> Dongjoon >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> On 2026-10-01 21:58 UTC Hyukjin Kwon wrote: >>>>>>>> > >> > >>>> > Hi all, >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > I've been running a trial of automated AI code review >>>>>>>> on new >>>>>>>> > >> > >>>> apache/spark >>>>>>>> > >> > >>>> > PRs, and I'd like to share how it works and get your >>>>>>>> feedback. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > For now the reviews are posted from my own GitHub >>>>>>>> account >>>>>>>> > >> > >>>> (@HyukjinKwon) >>>>>>>> > >> > >>>> > while I check that it behaves well. Once it's proven, >>>>>>>> I plan to >>>>>>>> > >> move >>>>>>>> > >> > >>>> it to >>>>>>>> > >> > >>>> > the https://github.com/spark-test account, so the >>>>>>>> reviews are >>>>>>>> > >> clearly >>>>>>>> > >> > >>>> > separate from my personal ones. I plan to run this >>>>>>>> for the next >>>>>>>> > >> week. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *What it does* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > Each review is a full, multi-pass review of the PR >>>>>>>> diff, run by >>>>>>>> > >> Claude >>>>>>>> > >> > >>>> > (via Claude Code). Findings are checked against the >>>>>>>> code before >>>>>>>> > >> > >>>> they're >>>>>>>> > >> > >>>> > posted. They're posted as a normal GitHub review: a >>>>>>>> short >>>>>>>> > >> summary plus >>>>>>>> > >> > >>>> > inline comments, each labeled with a severity, e.g. >>>>>>>> "Non-blocking >>>>>>>> > >> > >>>> (P2)" >>>>>>>> > >> > >>>> > or "Nit (P3)". Example: >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> >>>>>>>> > >> >>>>>>>> https://github.com/apache/spark/pull/59184#pullrequestreview-5379759787 >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *No LGTM, no approvals* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > - It never approves a PR and never posts "LGTM". >>>>>>>> Every review is >>>>>>>> > >> a >>>>>>>> > >> > >>>> plain >>>>>>>> > >> > >>>> > "Comment" review, so it doesn't count as an >>>>>>>> approval or a >>>>>>>> > >> change >>>>>>>> > >> > >>>> > request, and it has no effect on merging. >>>>>>>> > >> > >>>> > - If a review finds nothing new, it posts nothing at >>>>>>>> all: no "no >>>>>>>> > >> > >>>> > findings" or "all comments addressed" message. >>>>>>>> > >> > >>>> > - It doesn't push commits or edit anyone's PR. >>>>>>>> > >> > >>>> > - Human review and committer judgement are unchanged. >>>>>>>> Please >>>>>>>> > >> treat its >>>>>>>> > >> > >>>> > comments like any other reviewer's: feel free to >>>>>>>> push back or >>>>>>>> > >> > >>>> ignore. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *When a review is triggered* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > - A new PR is reviewed shortly after it's opened. >>>>>>>> Draft PRs are >>>>>>>> > >> > >>>> reviewed >>>>>>>> > >> > >>>> > once they're marked ready for review. >>>>>>>> > >> > >>>> > - After a review posts findings, the bot waits. It >>>>>>>> reviews again >>>>>>>> > >> only >>>>>>>> > >> > >>>> > when the author pushes new commits. >>>>>>>> > >> > >>>> > - It also waits while a PR has open review comments >>>>>>>> from anyone >>>>>>>> > >> that >>>>>>>> > >> > >>>> are >>>>>>>> > >> > >>>> > newer than the latest commit, so it doesn't pile on >>>>>>>> while the >>>>>>>> > >> > >>>> author is >>>>>>>> > >> > >>>> > still responding to reviewers. >>>>>>>> > >> > >>>> > - Replying in a thread without pushing doesn't >>>>>>>> trigger a >>>>>>>> > >> re-review. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *When it stops* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > - When a review finds nothing new. The PR gets a >>>>>>>> fresh review if >>>>>>>> > >> the >>>>>>>> > >> > >>>> > author pushes again later. >>>>>>>> > >> > >>>> > - When the PR is merged or closed. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *Early numbers* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > Over the first ~12 hours (Oct 1), it covered 10 PRs, >>>>>>>> including 5 >>>>>>>> > >> > >>>> > re-reviews after the author pushed. It posted 7 >>>>>>>> reviews with 14 >>>>>>>> > >> > >>>> findings >>>>>>>> > >> > >>>> > in total, mostly P3 nits. Four PRs came back clean >>>>>>>> and got no >>>>>>>> > >> comment. >>>>>>>> > >> > >>>> > Several authors pushed fixes, and the follow-up >>>>>>>> reviews found >>>>>>>> > >> nothing >>>>>>>> > >> > >>>> > new. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > *Feedback* >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > Please let me know if the comments are useful, noisy, >>>>>>>> or wrong, >>>>>>>> > >> either >>>>>>>> > >> > >>>> > here or on the PR. If you'd rather it not review your >>>>>>>> PRs, tell >>>>>>>> > >> me and >>>>>>>> > >> > >>>> > I'll exclude them. I'm also happy to adjust when it >>>>>>>> triggers >>>>>>>> > >> based on >>>>>>>> > >> > >>>> > what the community prefers. >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> > Thanks, >>>>>>>> > >> > >>>> > Hyukjin >>>>>>>> > >> > >>>> > >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> >>>>>>>> > >> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> > >> > >>>> To unsubscribe e-mail: [email protected] >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>> >>>>>>>> > >> > >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> > >> To unsubscribe e-mail: [email protected] >>>>>>>> > >> >>>>>>>> > >> >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe e-mail: [email protected] >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> Twitter: https://twitter.com/holdenkarau >>>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>>> Books (Learning Spark, High Performance Spark, etc.): >>>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>>> Pronouns: she/her >>>>> >>>> >>
