FYI After some distraction on the security front (and some learnings), I'm returning to triage. I learned a better approach during improveents of our security process. I basically reimplemented it - delegating much more (or rather almost everythign) to agents. I converted the previous tool and rules I came up with - to an agentic SKILL-only approach. And it seems to work much better—it's a bit slower and burns more tokens. But it's way more flexible and interactive and it pays for itself multiple times.
You can see the draft version here: https://github.com/apache/airflow/pull/65648 No code. Just an English description of the triage process: * Way easier to understand it * Self-descriptive and self-documented. * Way more conversational - you can just write "Trage PRs from this and that area opened in last week" - and it will do it. * Way easier to modify it * Self-learning: When you follow the triage it learns from your decisions and proposes updates to the triage process as you work on it * You can correct it (i.e. develop changes) and triage in the same interactive Claude session * Midterm: Much more suitable for 90% full automation (leaving only difficult cases for humans). It's pretty amazing what the agents can do - we are again down to about 450 or so PRs (many drafted), and I haven't done a full triage round yet. J. On Sat, Apr 4, 2026 at 12:19 PM Jarek Potiuk <[email protected]> wrote: > Very nice article from The New Stack explaining exactly why pre-review > triage is crucial > > There’s a hidden tax on every AI-generated merge request - The New Stack > https://share.google/DUF8FTW2FFcIzkXsZ > > J. > > On Fri, Apr 3, 2026, 16:47 Blain David <[email protected]> wrote: > >> The run of Github CoPilot on my PR's yesterday have certainly helped, as >> they already can pin-point a lot of issues otherwise the reviewer would >> have to do and also because I can see stuff we as human might not directly >> think of. Not all remarks are spot on, but most of them certainly are, so >> the ones that are not helpful I just downvote. So yes, that is a good thing >> to have imho. >> ________________________________ >> From: Jarek Potiuk <[email protected]> >> Sent: Friday, April 3, 2026 14:12 >> To: [email protected] <[email protected]> >> Subject: [DISCUSS] Current auto-triage stats/ learnigns and ask for >> maintainers >> >> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze >> niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, >> stuur deze e-mail als bijlage naar [email protected]<mailto: >> [email protected]>. >> >> *TL;DR: After a long triage session yesterday, I have a kind request for >> maintainers: if you could look at "ready for maintainer review" PRs in >> your >> areas, that would be great. * >> >> PRs marked "ready for maintainer review" have passed initial triage and >> require Your attention. The tool's main focus now is to move any pull >> requests that simply fail initial validation out of your view (Draft and >> Close). Also Kaxil ran CodePilot reviews on many of those "ready" ones >> (and some others) yesterday, and we want to see if that is helpful for >> reviews as well. >> >> I have been busy the last few weeks and it took longer than expected, but >> yesterday I finally completed a four-week loop of trying it and triaged >> all >> (!) 500+ opened PRs in about 4 hours. During this triage, 540 open PRs >> decreased to 492. The average triage time was about 1 minute per PR during >> a focused session. Many issues were skipped automatically because they do >> not need triage. Triage is only for issues from non-collaborators and for >> already triaged issues that might need some action (like Draft / Not >> responded / Closed). >> >> More details and instructions on how to provide feedback follow for >> interested parties. >> >> ------ >> >> *# Feedback* >> >> I would also love to hear specific feedback from maintainers, reviewers, >> and contributors. I created the #auto-triage-feedback >> < >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-airflow.slack.com%2Farchives%2FC0AQNS4DV2A&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf8cc9f6fcc724b0a4eca08de917a58e6%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639108151817628149%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=zMc2BpMkoYfFKeMEirHRtSMJ9QbWmpAjdlMKSOTP%2F3Q%3D&reserved=0 >> <https://apache-airflow.slack.com/archives/C0AQNS4DV2A>> - I do not >> promise >> to engage with all feedback if there is a lot, as the tools are in early >> stages, meaning there might be many issues and areas for improvement. At >> this stage let's gather the feedback and try to refine it so more people >> can use it regularly, and possibly we can automate it further. Or maybe >> even we will learn that it does not help at all, and only gets in the way. >> >> *# Findings so far* >> >> Some current findings (See the stats below): >> >> * We have about 80% of our PRs currently coming from external contributors >> (i.e. non-committers, non-collaborators) - that's a lot of work for >> maintainers >> >> * About 40% of the PRs marked as "done" are already merged (which is >> good), >> and most of those received responses and incorporated the triage comments. >> Which is cool. >> >> * About 60% of them were closed without being merged—some immediately, but >> mostly following this path: Draft -> Triage -> No response (more than 2 >> weeks) -> Close. This means those are really drive-by-contributors. >> >> * We have 127 PRs that seem ready for the "ready for maintainer review" >> label. It would be great if in your reviews of contributor issues in >> "your" >> areas you focus on those. >> >> *# Current status of the tool* >> I have not yet asked others to participate much yet, but if anyone wants >> to >> try it, feel free to start using it - with `breeze pr auto-triage >> --reviews-for-me`. This will only select issues where CODEOWNERS >> automatically sets you as a reviewer or where you are mentioned. >> >> But for that, >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64669&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf8cc9f6fcc724b0a4eca08de917a58e6%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639108151817644852%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6qoQkbB8WBJtS%2FjfyeGAeSBb7hVPHdYMB8h2Evi2iE4%3D&reserved=0 >> <https://github.com/apache/airflow/pull/64669> will have to be >> merged. Yes, a lot of changes and tweaks have accumulated—sorry for such a >> huge PR. This will likely finally stabilise and I will refactor the >> algorithms, split them into smaller pieces, and then we can proceed with >> more incremental updates. Next week I am on a PyCon LT conference but I >> will focus mostly on incremental triaging and tweaking. >> >> It includes cumulative learning from about 20 smaller triage sessions I've >> done in the past weeks. I also have a few things to add after yesterday's >> longer session, specifically cleaning up the algorithmic choices to better >> determine default actions. >> >> The tool is not perfect yet, and requires careful choices especially since >> we still have many flakes. I had to do more manual assessment than I would >> like to - I hope we can stabilise them after 3.0.0 release. And make it >> more useful and I hope it will be ready for others to participate. I am >> also going to look at the responses—I guess in some cases the triage was >> "unfair," and I am trying to optimise it. It's still far from full >> automation; it requires close human supervision (as expected at this >> stage). >> >> I am iterating **fast** on it - learning with every triage run while also >> doing other things as well. I will try to make it really simple to follow. >> We have a TUI mode that is good for testing and debugging (and possibly >> later for a focused review mode - which we already have but it's not as >> useful) - but I found the CLI mode far more useful overall. TUI is far too >> much of a distraction - but might be cool if you want to focus on smaller >> groups of PRs to review and later can help with review- and we have Andre >> Ahlert who already contributes some nice improvements there. >> >> *# Stats* >> >> I've also built the `pr stats` command: >> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F64667&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf8cc9f6fcc724b0a4eca08de917a58e6%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639108151817656613%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=SfmU4QfGqpwlCgzc5jKZDAzRQqoBXXqmPRTMyy23p8k%3D&reserved=0 >> <https://github.com/apache/airflow/pull/64667> - happy to receive >> reviews, >> and this stats command still needs some tweaking and improvement, which >> will follow. >> >> I have also built stats and track the current status of triaged >> collaborator PRs. >> >> *## Triaged "final" state:* >> >> In short 40 out of 102 have already been merged after responding to >> triage, >> 62 have been closed without merging (no response on triage). >> >> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fibb.co%2FLz2pj259&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf8cc9f6fcc724b0a4eca08de917a58e6%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639108151817668047%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=BrQVxG5pG%2BX2rIp9WjKkMuaXNBTdmayVYSCZWXVoTZY%3D&reserved=0 >> <https://ibb.co/Lz2pj259> - image was too large to attach >> >> *## Current open PRs status* >> >> * As of yesterday we had 492 open PRS >> * 400 of those are contributor PRs >> * 126 of those are "ready for maintainer review" >> * 200 of those are already drafted and triaged, waiting for the >> contributor's response (128) or they are simply unfinished drafts. >> >> Those stats will change daily - and there might be some missing things >> there that I will track and add any missing items over the coming days >> (After Easter). >> >> >> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fibb.co%2FNHzdrQx&data=05%7C02%7Cdavid.blain%40infrabel.be%7Cf8cc9f6fcc724b0a4eca08de917a58e6%7Cb82bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C639108151817682501%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kgJadYL2TrXnuBqCEykT7OUxchdNX1GWPfcxqfxseHA%3D&reserved=0 >> <https://ibb.co/NHzdrQx> - - image was too large to attach >> >> Also - if you have general feedback and comments to it - feel free. >> >> I will pick it up after Easter - and for those who celebrate Easter, have >> a >> happy, AI-free, family-focused one. >> >> I certainly plan it this way. >> >> J. >> >
