RoyLee1224 opened a new pull request, #69308: URL: https://github.com/apache/airflow/pull/69308
## What Adds an eval harness (`dev/skill-evals/`) for testing AGENTS.md guidance against real scenarios. It answers: "does my AGENTS.md change actually affect agent behavior?" ```bash uv run dev/skill-evals/eval.py --repeat 3 ``` Compares the `main` branch AGENTS.md against your working tree. Each arm is a git worktree with the full repo, so the agent sees real source files. If AGENTS.md is unchanged, the working arm is skipped automatically. No API key needed — authenticates via Claude Code OAuth (`claude /login`). ## Demo: testing the newsfragment golden rule Following [the discussion in #release-management](https://apacheairflow.slack.com/archives/C03G9H97MM2/p1782814415838669) Tested the golden rule (#67982) against real PRs where reviewers asked to remove newsfragments: | Case | with AGENTS.md | without | |------|:-:|:-:| | Provider pod leak [#67333](https://github.com/apache/airflow/pull/67333#discussion_r3292590738) | 3/3 ✓ | 0/3 | | API optimization #66696 | 3/3 ✓ | 1/3 | | Scheduler fix [#64322](https://github.com/apache/airflow/pull/64322#discussion_r3032123197) | 2/3 | 0/3 | | i18n cache fix [#65720](https://github.com/apache/airflow/pull/65720#discussion_r3138464983) | 1/3 | 0/3 | Golden rule works for clear cases but struggles with ambiguous fixes where the model reasons "this bug affects users." **Open question:** I'm not sure the current case selection is the right design. Would appreciate maintainers' thoughts on what cases are worth covering. ### cli demo <img width="2844" height="1564" alt="CleanShot 2026-07-02 at 00 57 34@2x" src="https://github.com/user-attachments/assets/2c226802-936a-41d7-bd7d-eac96a248785"; /> ### UI demo(run `npx [email protected] view`): <img width="3024" height="1802" alt="ui demo" src="https://github.com/user-attachments/assets/07cdcb5c-7fb1-405b-b769-07104af364a3"; /> ## How it works 1. Creates git worktrees — one with `main`'s AGENTS.md, one with your working tree version. Both are full repo checkouts. 2. Generates a [promptfoo](https://github.com/promptfoo/promptfoo) config with `anthropic:claude-agent-sdk` provider and structured JSON output. 3. Runs each case against all arms in parallel, reports diff. 4. Worktrees cleaned up on exit. ## Files ``` dev/skill-evals/ eval.py # entry point (Python, per dev/ guidelines) cases/newsfragment.yaml # cases from real PRs (#64322, #65720, #66696, #67333) README.md # setup and usage .pre-commit-config.yaml # prek hook: remind to run eval on AGENTS.md changes scripts/ci/prek/check_eval_reminder.py ``` ## Future use - Auto-generate a summary table (like the one in this PR) from promptfoo results. Might be useful for pasting into PRs that change AGENTS.md; open to discussion - As models improve, some guidance may become unnecessary — the eval helps determine which rules the model still depends on and which can be safely removed - Add cases for routing rules that matter — the prek hook reminds contributors to run the eval when AGENTS.md changes - Use the eval to validate moving guidance from AGENTS.md into skills — run before and after to confirm no regression - Currently tests Claude only; the architecture (promptfoo + structured output) could extend to other agent runtimes <!-- SPDX-License-Identifier: Apache-2.0 https://www.apache.org/licenses/LICENSE-2.0 --> <!-- Thank you for contributing! Please provide above a brief description of the changes made in this pull request. Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/ Please make sure that your code changes are covered with tests. And in case of new features or big changes remember to adjust the documentation. For user-facing UI changes, please attach before/after screenshots (or a short screen recording) so reviewers can assess the visual impact. Feel free to ping (in general) for the review if you do not see reaction for a few days (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications. In case of an existing issue, reference it using one of the following: * closes: #ISSUE * related: #ISSUE --> --- ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [x] Yes, Claude Code (Opus 4.8) <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
