Hi everyone, I’ve spent some time reflecting on all the great points raised here. Our shared goals are to ensure human ownership and review, keep agents as helpful assistants rather than sole authors, and reduce the cognitive load from long AI-generated descriptions.
I really like Shahar's proposal and would love to build on it with a few suggestions to make our expectations clear and supportive for our human contributors: - Explicit Instructions: Let’s be very open in our templates and AGENTS.md. We can instruct agents to pause and ask the human to write the description, noting that this personal touch is essential for the PR to stay open. - Human Review Checkbox: I suggest adding a checkbox: "- [ ] I have reviewed this code myself." We’ll instruct agents to leave this for the human to check, ensuring that vital moment of reflection. - Instead of copy-pasting (which I find awkward), we can instruct the agents to use `gh --web`, `--template` (to include the template), and `--draft` (following Pierre's idea). This creates natural checkpoints—filling the description, checking the box, clicking submit, and undrafting—that encourage human involvement. We should also state the consequences for non-compliance: To keep our queue healthy, we should use an automated process to close PRs that miss these steps, with a note explaining how to resubmit them with human input. All those expectations and closing etc. should be equally applied to all PRs, including maintainer PRs. This will also allow those of us who use agents to monitor the process and refine the instructions if we see any loopholes that agents try to bypass or learn how to circumvent. This will allow us to continuously improve the process. I believe this approach balances productivity with the high-quality human collaboration we all value. What do you think? Best regards, Jarek On Tue, Jun 9, 2026 at 5:00 PM Shahar Epstein <[email protected]> wrote: > Here's a more concrete suggestion: > > Updating the PR template in such a way that: > 1. Human summary is now a MUST - at least a oneliner* (or more, depending > on the scope - TBD) that describes the suggested changes written by the > PR's author themselves (without AI assistance). > 2. AI summary is optional. However, when included - it MUST be bound within > a collapsible box, mainly to save cognitive load for maintainers and > contributors, but also to encourage human interaction like we used to do > before it all started. > 3. PR's author (human) should be the one declaring the AI usage checkbox - > added a short statement of ownership. > > Contributors will be instructed to use this template and adhere to the > instructions when creating a PR. > Agents may push branches to forks, but they will be instructed to avoid > creating PRs on their own to the upstream repository, and instead provide > the link for creating the PR using this template (they could suggest an AI > summary, but the contributor should copy and paste it manually to the > collapsible box). Trying to work around that might result in an M&M test > directly in the PR's description (TBD). > > Example is available here <https://github.com/apache/airflow/pull/68055> - > I've made HTML comments visible, they will be hidden in the real thing. > > Took inspiration for this idea from https://tenbluelinks.org/ , that hides > the AI overview on Google if you're not interested (highly-recommended > btw). > > Can we live with that? > > > Shahar > > On Tue, Jun 9, 2026 at 3:30 PM Ash Berlin-Taylor <[email protected]> wrote: > > > I don’t care one way or another about using AI as a tool in CI, that is > > secondary to my goal which is to try and do something to make it clear > what > > we expect from people wanting to contribute to Airflow, namely: > > > > 1. Human involvement. > > > > By submitting a PR you are saying “yes I want to be a member of the > > community”. Agents submitting without human interaction go against this. > > > > 2. Human ownership. > > > > It is _your responsibility_ as the PR author to follow up on it, address > > comments, and request reviews. > > > > > > I frankly find the AI generated triage comments verbose, and a waste of > > time and pure noise even without the `@` spam. > > > > If the user doesn’t care enough about their own PR to follow up on it: > > close it after some time. We don’t need to baby sit them. Nor do I need > yet > > more commit email messages to read through. > > > > > > So how does it sound: It sounds like hell to me and an even bigger waste > > of electricity in a climate crisis. > > > > I want to be involved in a community of humans working to build software. > > I do not want to see LLMs producing so much output that other people need > > LLMs to summarise it, with no humans looking at things. > > > > -ash > > > > > On 9 Jun 2026, at 13:18, Jarek Potiuk <[email protected]> wrote: > > > > > >> Why? Because AI “instructions” cannot be trusted. And I am after a > > signal > > > that people are blindly using LLMs without enough human introversion. > > > > > > But is not that what you are doing? This proposal is about adding > another > > > AI instruction (just hidden in HTML) - how is that going to help? > > > > > >> You already updated the instructions to not `@` the reviewer here > > > > > > Indeed, LLMs are not deterministic by nature. But they are improvable. > > > Through iterations of refinement and adding more guardrails we can > > improve > > > it—and this is exactly why I am running it manually to make it better. > > This > > > is the same as in regular breeze development in the past. Initially, > > there > > > were many small issues - and I remember how you complained about them > and > > > how unnecessary they seemed—yet we now perfected it over time. Now, it > > > allows all contributors and maintainers to work much more efficiently > and > > > lose less time. BTW. Thanks for notifying me; I must strengthen this > one > > > and see why, as there might be another improvement to implement. This > is > > > also why we are not "yet" doing CI analysis by AI - because I want to > > > iterate on it and fix it in the way to know which parts are > > deterministic. > > > > > >> I want to do anything and everything to reduce the drive by > contribution > > > with no human activity. I’m happy to spend my time helping humans, but > if > > > they are just going to feed that back to an LLM and burn an egregious > > > amount of carbon: no thank you. > > > > > > And again I am not sure how the proposal to add that instruction would > > > address this particular issue? Are you just proposing to add another > > > instruction for the LLM (or am I wrong?). How does it solve the > problem? > > > > > > From what I understand we have two basic proposals here - that > contradict > > > each other: > > > > > > * Ash - do not use AI to fight with AI at all > > > * Amoght, Shahar - use AI in CI > > > > > > But I think, the triage I am running now shows a third way: > > > > > > * we use AI to try out and generate triage action and figure out which > > > parts are practically 100% deterministic and can help with triage (this > > is > > > the stats I am gathering now) > > > * qe use AI to convert the SKILLS we have into deterministic CI code > that > > > does those triage steps (no AI used at all at runtime) > > > * we continue perfecting the manually-triggered AI SKILLS to get more > AI > > > heuristics that we can turn into deterministic CI code > > > > > > This seems to fulfill seemingly contradictory expectations that > different > > > people have in a nice way. I am about to produce stats from the last > run > > > and was just about to propose this approach. > > > > > > How does it sound Ash, Amogh, Shahar and others ? > > > > > > J. > > > > > > > > > On Tue, Jun 9, 2026 at 12:55 PM Ash Berlin-Taylor <[email protected]> > > wrote: > > > > > >> Why? Because AI “instructions” cannot be trusted. And I am after a > > signal > > >> that people are blindly using LLMs without enough human introversion. > > >> > > >> Want a prime example? > > >> > > >> The pr triage skill. > > >> > > >> You already updated the instructions to not `@` the reviewer here > > >> > > > https://github.com/apache/airflow-steward/blob/76cfa5e1d2e682b88df5205e9cda396df51a66b6/skills/pr-management-triage/comment-templates.md#reviewer-mention-policy > > >> > > >>> When a comment's only addressee is the PR author (the > > >> request-author-confirmation, reviewer-ping author-primary, and > > review-nudge > > >> author-primary templates), the body references the reviewer without > > >> @-mentioning them > > >> > > >> And yet the LLM did it again: > > >> https://github.com/apache/airflow/pull/66633#discussion_r3344849352 > > >> > > >>> @korex-f — A reviewer (@ashb) has requested changes on this PR, so > I've > > >> removed the ready for maintainer review label — the next step is on > your > > >> side. Could you address the review comments (push a fix, or reply > > in-thread > > >> explaining why the feedback doesn't apply)? Once addressed, re-request > > >> review from @ashb or re-mark the PR ready and it returns to the > > maintainer > > >> queue. Thank you. > > >> > > >> And frankly I’m tired of all this shit. > > >> > > >> I want to do anything and everything to reduce the drive by > contribution > > >> with no human activity. I’m happy to spend my time helping humans, but > > if > > >> they are just going to feed that back to an LLM and burn an egregious > > >> amount of carbon: no thank you. > > >> > > >> -ash > > >> > > >> > > >>> On 9 Jun 2026, at 10:38, Jarek Potiuk <[email protected]> wrote: > > >>> > > >>> Hi Ash, Amogh, and Shahar, > > >>> > > >>> Ash, I'm curious to learn more about how the "brown m&m test" differs > > >> from > > >>> our current request for agents to identify themselves. Could you help > > me > > >>> understand the flow and the specific benefits you see? It feels > similar > > >> to > > >>> me, but I'd love to hear your perspective in case I'm missing a > nuance. > > >>> > > >>> Regarding the gh pr create --web approach, we included those > > instructions > > >>> to ensure we meet ASF legal guidelines for Gen-AI headers, and to > > support > > >>> contributors who might not have Copilot. That said, if you have ideas > > on > > >>> how to trim the context or improve the templates, we truly appreciate > > PRs > > >>> that improve them—and many people already have. AGENTS.md is a team > > >> effort, > > >>> and we’re always looking for ways to make it better. Let's keep our > > >>> collaboration positive as we refine these processes together. > > >>> > > >>> Amogh and Shahar, yep the idea of an validatio step in the CI for > > >>> first-time contributions is something we should implement sooner or > > >> later. > > >>> I have actually been gathering stats on this for the last two weeks. > > I’ve > > >>> been preparing to see how manually triggered triage tasks can turn > into > > >>> automated ones—I'm gathering stats on when human judgment is needed. > I > > >>> shared some stats about this recently and will continue gathering > them. > > >> The > > >>> next step is discussing here what and how we can automate. > > >>> > > >>> Also, the current triage process already uses our Pull Request > criteria > > >> to > > >>> pre-classify the PRs and only marks them with "ready for maintainer > > >> review" > > >>> if those criteria are met. So, if there are any specific criteria > you’d > > >>> like to see added to our "Pull request criteria," PRs are most > welcome > > >>> there as well. > > >>> > > >>> Best regards, > > >>> > > >>> Jarek > > >> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
