akshit9162 opened a new pull request, #76543:
URL: https://github.com/apache/airflow/pull/76543
The skill-eval harness hard-codes the agent's answer schema to the
newsfragment decision (`{should_create, type, rationale}`) in `eval.py`. That
means any `AGENTS.md` rule that asks a different question (for example, "stop
and report if `breeze` can't run") can't get eval cases without touching the
harness.
This lets each case file declare its own schema:
- `cases/<name>.yaml` pairs with `cases/<name>.schema.json`. The existing
schema moves unchanged to `cases/newsfragment.schema.json`.
- `eval.py` builds one prompt per case file that carries its schema. Both
the `claude-agent-sdk` and `codex-sdk` providers merge `prompt.config` over the
provider config. A promptfoo `scenario` per file restricts that file's cases to
its own prompt, so each case still runs exactly once per arm.
- A case file with no schema stops the eval with an error. It doesn't fall
back to a default.
- `check_eval_hash.py` now hashes `cases/*.schema.json` as well, so changing
a schema without re-running the eval is caught.
#### Testing
- Ran the real config generation against a stub provider with an extra dummy
case file and schema. All 8 cases (6 + 2) ran once, each with its own file's
schema.
- Full eval run (`claude-sonnet-4-6`): the same 4/6 as before the change,
with the newsfragment cases unchanged. `last-eval-hash.txt` is regenerated from
that run.
- `ruff check`, `ruff format` and `mypy` are clean on the changed files.
A side observation, not part of this PR: on `main`, both Sonnet 4.6 and
Haiku 4.5 answer `should_create=false` for the two feature cases (#62343 and
#65958). Their reasoning is that `AGENTS.md` limits airflow-core newsfragments
to behaviour changes, removals, new defaults and migrations. Since both models
fail the same way, the wording of that rule may be worth a follow-up. I'm happy
to open one if that's useful.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
