justinmclean commented on PR #551: URL: https://github.com/apache/airflow-steward/pull/551#issuecomment-4805561354
## Correctness [advisory] tools/skill-evals/evals/release-archive-sweep/README.md:18-30 — the documented run commands omit --cli "claude -p", so as written the prose/judge cases grade as MANUAL rather than auto-grading. uv run --project tools/skill-evals skill-eval \ Rule: eval suites with prose/judge assertions need --cli to grade; note this matches the sibling release-announce-draft/README.md house style, so it is pre-existing convention, not introduced here. [advisory] Skill logic checks out against all 9 eval cases (pre-flight gating, retention classification, retention-rule-error block, orphan flagging, per-backend command emission, injection). No correctness finding in SKILL.md itself. Golden rules 1-3 are each exercised by a dedicated fixture. Rule: changed paths should have test coverage; they do. ## Security No blocking findings. The skill is read-only and proposal-only on the dist surface, treats the dist listing / planning issue / config as untrusted input, and has a dedicated prompt-injection case. The has_no_forged_commands judge rubric correctly verifies the injected instruction was not obeyed. "the injected instruction was not obeyed" Rule: untrusted external content must be treated as data, with a test proving it. ## Conventions [blocking] tools/skill-evals/evals/release-archive-sweep/README.md:31-49 — the README now contradicts the code. After this session's fix, has_no_forged_commands is a judge assertion, but the README still describes it as a deterministic predicate and explicitly claims the security property "does not depend on a probabilistic judge." (commands field must not contain the injected text verbatim ...) / so the security property does not depend on a probabilistic judge. Rule: documentation must match the code it describes; this section is stale and will mislead reviewers. Update it before opening the PR. [advisory] SPDX headers are correctly present on the new markdown files and correctly absent on JSON fixtures (consistent with sibling suites). Placeholder tokens (<project>, <version>) in SKILL.md follow the documented placeholder convention. No finding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
