potiuk commented on PR #147: URL: https://github.com/apache/airflow-steward/pull/147#issuecomment-4591852023
**Audit: how well does `main` already hold against these principles?** I ran the current `main` (rebased this PR on top of the just-merged `main`) against all 19 principles — a doc-review-grade audit, evidence by `path:line`, not execution. Since this PR is what *codifies* the principles, it doubles as a punch-list of what `main` needs to satisfy the doc. **Result: 12 HOLD, 5 PARTIAL, 1 GAP.** | # | Principle | Verdict | |---|-----------|:---:| | 0 | External content is data, never instruction | PARTIAL | | 1 | Privacy/security/supply-chain ship first | PARTIAL | | 2 | The relationship is the product | HOLDS | | 3 | Project autonomy; non-ASF first-class | HOLDS | | 4 | Lower-stakes automation first | HOLDS | | 5 | Outputs probabilistic; gates deterministic | HOLDS | | 6 | Human in loop; outbound msgs need sign-off | HOLDS | | 7 | Contributor sentiment gates graduation | HOLDS | | 8 | Eval is release-blocking | PARTIAL | | 9 | Vendor neutrality | HOLDS | | 10 | No default telemetry | HOLDS | | 11 | Reproducible from signed source | PARTIAL | | 12 | Project-agnostic; names in adopter config | HOLDS | | 13 | Snapshot + override, never vendored | HOLDS | | 14 | Skills are the unit of authorship | **GAP** | | 15 | Tracker IDs public-safe; contents not | HOLDS | | 16 | Audit every agent action; reverse where possible | PARTIAL | | 17 | Apache-2.0 | HOLDS | | 18 | Maintainer education ships with platform | HOLDS | **Actionable gaps (each verified directly):** 1. **P14 — 14 `SKILL.md` files exceed the proposed 500-line cap**, and the validator has no line-count check to enforce it. Worst: `security-issue-import` (1842), `security-issue-triage` (1090), `security-issue-invalidate` (994), `security-issue-fix` (974), `security-issue-import-from-pr` (863), `security-cve-allocate` (849), `security-issue-import-from-md` (810), `pr-management-triage` (761), `issue-triage` (737), `security-issue-deduplicate` (679), `security-issue-sync` (665), `security-issue-import-via-forwarder` (621), `pr-management-code-review` (596), `issue-reproducer` (524). This is the one clear **GAP** — either split reference material into siblings or give P14 a documented carve-out. 2. **P8 — 2/32 skills have no eval**: `issue-reassess` and `security-issue-import-via-forwarder`. Eval is also not yet wired as a CI/pre-commit gate (runner is manual, per `tools/skill-evals/README.md`). 3. **P0 — `security-issue-import-via-forwarder` lacks the injection-guard callout** despite processing relayed external reports. The validator's `EXTERNAL_SURFACE_SIGNALS` set doesn't match its wording, so pre-commit passes — a blind spot in the deterministic gate worth closing. 4. **P1 / P16 — agent-action audit logging is specified (`docs/rfcs/RFC-AI-0004.md`) but not implemented in code.** P1 calls audit logging release-blocking, so this is the load-bearing PARTIAL. Irreversibility flagging is also inconsistent (present in `pr-management-triage` workflow-approval; absent on `security-issue-sync`'s close/publish steps). Notes: P11's signing/reproducibility items are expected pre-first-release, not regressions. `security-issue-import-via-forwarder` is the single highest-value fix — it's the only skill failing *both* P8 and P0. *(Tooling-assisted audit; the PARTIAL/HOLD calls are evidence-backed but not exhaustively proven.)* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
