YAshhh29 commented on PR #76784: URL: https://github.com/apache/airflow/pull/76784#issuecomment-6751980001
How I got here, and the local runs behind the description. Everything below is on Linux (WSL Ubuntu) at ee13b495f7. **Where it started** This note in the skill-evals README on `main` (at this branch's base) is what I was reading after #73254: https://github.com/apache/airflow/blob/19d80a886bf1eb990d119658fee48c2234d28e1a/dev/skill-evals/README.md#L187-L190 **The gate, after this PR** Appending a line to the skill's SKILL.md fails the check and names the suite. Reverting it passes again: ```console $ echo '<!-- gate test -->' >> .agents/skills/airflow-pr-draft-summary/SKILL.md $ prek run check-eval-hash --all-files Verify skill-eval ran on current AGENTS.md, skills and cases.......................Failed - hook id: check-eval-hash - exit code: 1 The airflow-pr-draft-summary skill or its cases changed without re-running its skill-eval (dev/skill-evals/skills/airflow-pr-draft-summary/last-eval-hash.txt does not match). Run: SKILL_NAME=airflow-pr-draft-summary prek run run-skill-eval --hook-stage manual --all-files then commit the updated dev/skill-evals/skills/airflow-pr-draft-summary/last-eval-hash.txt WIP commit: SKIP=check-eval-hash git commit ... Can't run the eval? Ask a maintainer to run it and push the updated hash file to your PR branch (see dev/skill-evals/README.md). $ git checkout -- .agents/skills/airflow-pr-draft-summary/SKILL.md $ prek run check-eval-hash --all-files Verify skill-eval ran on current AGENTS.md, skills and cases.......................Passed $ git status On branch skill-evals-skill-cases Your branch is up to date with 'origin/skill-evals-skill-cases'. nothing to commit, working tree clean ``` Appending a line to `cases/titles.yaml` or a newline to `output-schema.json` gives the same failure. Changing AGENTS.md fails only the existing top-level check, not the suite: ```console $ echo '<!-- gate test -->' >> AGENTS.md $ prek run check-eval-hash --all-files Verify skill-eval ran on current AGENTS.md, skills and cases.......................Failed - hook id: check-eval-hash - exit code: 1 AGENTS.md or dev/skill-evals/cases/ changed without re-running the skill-eval (dev/skill-evals/last-eval-hash.txt does not match). Run: prek run run-skill-eval --hook-stage manual --all-files then commit the updated dev/skill-evals/last-eval-hash.txt WIP commit: SKIP=check-eval-hash git commit ... Can't run the eval? Ask a maintainer to run it and push the updated hash file to your PR branch (see dev/skill-evals/README.md). ``` **Tests, static checks, merge** ```console $ uv run --project scripts pytest scripts/tests/ci/prek/test_check_eval_hash.py -q ...................... [100%] 22 passed in 0.12s $ SKIP=lychee-docker prek run --from-ref 19d80a886b --to-ref HEAD Run 'ruff' for extremely fast Python linting.......................Passed Run 'ruff format'..................................................Passed Verify skill-eval ran on current AGENTS.md, skills and cases.......Passed Check YAML files with yamllint.....................................Passed Run markdownlint...................................................Passed Run codespell......................................................Passed Run mypy for dev...................................................Passed Run mypy for scripts...............................................Passed ``` 53 hooks passed (the list above is trimmed). `update-breeze-cmd-output` failed only because breeze isn't installed here ("The `breeze` command is not on path"), and lychee needs Docker. Both are left to CI. ```console $ git merge-tree --write-tree --name-only 8930bc3db2 ee13b495f7 cdcd0942b8dbde290bd6f8d29f85834b99ca43a0 ``` No conflicts with main at 8930bc3db2. **The eval run** `eval-aB0-2026-11-02T18:31:08` (`working` = the branch's AGENTS.md, `main` = my local `main`'s, a week older, same SKILL.md, 3 runs per case), filtered to the failing runs. With the filter on, the red badge counts only the filtered rows, so it reads 0%; the real totals (16/18 and 13/18) are next to it: <img width="1877" height="877" alt="image" src="https://github.com/user-attachments/assets/f7af5651-3ca1-45d5-a7be-01e05051d4d8"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
