RoyLee1224 commented on PR #69861:
URL: https://github.com/apache/airflow/pull/69861#issuecomment-4998599293
I was curious what a wrong documented command like this actually costs an
agent in tokens, so I ran this fix through the skill-eval harness (#69308) as a
before/after A/B — same task ("determine which CI test types run for HEAD"),
same tools, only AGENTS.md differs, n=3 per arm:
<img width="1916" height="868" alt="CleanShot 2026-07-17 at 10 44 03@2x"
src="https://github.com/user-attachments/assets/25d4c6ce-f278-4052-acc9-6d433eec3fcf";
/>
Every "before" run trusted the doc, ran `breeze selective-checks`, failed,
and had to rediscover `breeze ci selective-check` via `--help`. So the wrong
command roughly doubles the token cost of every affected task — and that's a
floor: agents re-pay it on every fresh session, and in a real dev session each
wasted turn re-sends the full context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
