This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-steward.git
The following commit(s) were added to refs/heads/main by this push:
new fde4cc3b docs(spec): add repo-health-family spec and design docs (#558)
fde4cc3b is described below
commit fde4cc3ba3a7a6a6107c4574e8ab2173e1a2bb09
Author: Justin Mclean <[email protected]>
AuthorDate: Sat Jun 27 06:18:02 2026 +1000
docs(spec): add repo-health-family spec and design docs (#558)
Formalises the repo-health audit family, closing the gap identified in
triage-mode.md Known Gaps ('Repo-health audits are a one-off with no
family around them').
- tools/spec-loop/specs/repo-health-family.md: new functional spec
(status: experimental, mode: Triage). Defines the read-only or
propose-then-confirm contract all family members must follow, lists
ci-runner-audit as the sole shipped member, and names four candidates:
workflow-security-audit (zizmor-backed), dependency-audit,
license-compliance-audit, and flaky-test-triage.
- docs/repo-health/README.md: design doc and family overview. Describes
each candidate skill scope, rule classes, and adopter-config keys;
sketches the planned projects/_template/repo-health-config.md contract
so future build iterations have a clear shape to implement against.
Validation: spec-validate reports no violations; skill-and-tool-validate
passes (soft asf-coupling warnings are pre-existing, not introduced here).
Generated-by: Claude (Opus 4.7)
---
docs/repo-health/README.md | 176 ++++++++++++++++++++++++++++
tools/spec-loop/specs/repo-health-family.md | 116 ++++++++++++++++++
2 files changed, 292 insertions(+)
diff --git a/docs/repo-health/README.md b/docs/repo-health/README.md
new file mode 100644
index 00000000..3c98a82c
--- /dev/null
+++ b/docs/repo-health/README.md
@@ -0,0 +1,176 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update
-->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents** *generated with
[DocToc](https://github.com/thlorenz/doctoc)*
+
+- [Repo-health audits — family overview](#repo-health-audits--family-overview)
+ - [Current skills](#current-skills)
+ - [`ci-runner-audit` (experimental)](#ci-runner-audit-experimental)
+ - [Candidate skills (not yet built)](#candidate-skills-not-yet-built)
+ - [`workflow-security-audit` (proposed)](#workflow-security-audit-proposed)
+ - [`dependency-audit` (proposed)](#dependency-audit-proposed)
+ - [`license-compliance-audit`
(proposed)](#license-compliance-audit-proposed)
+ - [`flaky-test-triage` (proposed)](#flaky-test-triage-proposed)
+ - [Adopter contract (planned)](#adopter-contract-planned)
+ - [Cross-references](#cross-references)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+# Repo-health audits — family overview
+
+Read-only agent-assisted audits that surface repository maintenance signals
+a human would otherwise have to detect by hand: runner label obsolescence,
+Actions workflow security issues, stale or vulnerable dependencies,
+license/NOTICE drift, and flaky-test patterns. Every skill in this family
+produces a human-readable report and proposes remedies; applying any change
+is the maintainer's action.
+
+The family lives under `mode: Triage` in the framework taxonomy — the same
+classify-and-propose discipline the security and PR-management triage skills
+follow. See [`docs/modes.md` § Triage](../modes.md#triage).
+
+---
+
+## Current skills
+
+### `ci-runner-audit` (experimental)
+
+Reads every GitHub Actions workflow file across one repo, a named set, one
+Apache project's repos, or the full Apache GitHub org and surfaces two
+classes of issue:
+
+1. **Obsolete runner labels** — `ubuntu-18.04`, `ubuntu-20.04`, `windows-2019`,
+ and other GitHub-deprecated hosted-runner label strings that silently fall
+ back to a later image or will soon break.
+2. **macOS architecture mismatches** — a workflow step targeting an `arm64`
+ macOS runner that invokes an `x86_64`-only tool or vice versa.
+
+Output is a markdown audit report grouped by repo and by issue class.
+Read-only; no workflow files are modified.
+
+---
+
+## Candidate skills (not yet built)
+
+These are enumerated from the triage-mode.md Known Gaps. Each will become
+its own build item once the family shape is confirmed through
+`ci-runner-audit` pilot evaluations.
+
+### `workflow-security-audit` (proposed)
+
+Run [`zizmor`](https://woodruffw.github.io/zizmor/) — the GitHub Actions
+security scanner already wired into the framework's own pre-commit suite —
+across one repo or a named set and surface findings for human review.
+
+Proposed finding classes:
+
+- Injection vulnerabilities (`run:` steps using `${{ github.event.* }}` or
+ `${{ github.head_ref }}` in untrusted contexts)
+- Excessive permissions (`permissions: write-all` or unnecessary `write`
+ scopes on workflow-level or job-level grants)
+- Unpinned external actions (floating `@main`, `@master`, or tag-only
+ references instead of a commit SHA)
+- Self-hosted runner scope leakage (secrets available to PRs from forks)
+
+Each finding class maps to a concise prose description and a suggested
+remediation (scope reduction, SHA pinning, `${{ env.SAFE_VAR }}` pattern).
+
+**Adopter contract**: reads `<project-config>/repo-health-config.md`
+(planned) for which rule classes to enable and which repos to audit.
+
+### `dependency-audit` (proposed)
+
+Check direct and transitive dependencies for known vulnerabilities (via
+`pip-audit` / `npm audit` / `trivy` depending on the project's language
+stack) and surface those that have available patches. One finding per
+dependency, formatted for maintainer triage. Does not open update PRs
+autonomously — proposes one per affected dependency.
+
+**Adopter contract**: reads `<project-config>/repo-health-config.md` for
+the dependency manager and audit tool to use.
+
+### `license-compliance-audit` (proposed)
+
+Verify that:
+
+1. Every source file under a configured path carries the project's required
+ SPDX-License-Identifier header line.
+2. The `LICENSE` file matches the declared SPDX expression.
+3. The `NOTICE` file lists every bundled dependency that its license
+ requires to appear in attribution notices.
+
+Surfaces discrepancies as a structured report (file path, issue class,
+suggested correction) without modifying any file.
+
+**Adopter contract**: reads `<project-config>/repo-health-config.md` for
+the required SPDX expression and which source paths to audit.
+
+### `flaky-test-triage` (proposed)
+
+Parse GitHub Actions run history for a named repo over a configurable window,
+compute per-test failure rates (differentiating consistent failures from
+intermittent ones), and produce a prioritised triage list: tests failing
+above a configurable threshold that are likely flaky rather than
+deterministically broken.
+
+Signals used: run outcome (`success` / `failure`), re-run count on the same
+SHA, test-name patterns across runs. No test code is modified.
+
+**Adopter contract**: reads `<project-config>/repo-health-config.md` for
+the audit window, the failure-rate threshold, and which test-name patterns
+to include or exclude.
+
+---
+
+## Adopter contract (planned)
+
+A future `projects/_template/repo-health-config.md` will declare per-skill
+switches:
+
+```yaml
+repo_health:
+ ci_runner_audit:
+ # Runner label families to flag. Defaults to the GitHub-deprecated list.
+ deprecated_runner_labels: [ubuntu-18.04, ubuntu-20.04, windows-2019]
+ # Repos to audit; leave empty to audit only the project's own repos.
+ extra_repos: []
+
+ workflow_security_audit:
+ # zizmor rule classes to enable (all enabled by default).
+ enabled_rules: [injection, excessive-permissions, unpinned-actions,
fork-secrets]
+
+ dependency_audit:
+ # Dependency manager(s) in use. Selects the audit tool adapter.
+ # Allowed values: pip, npm, cargo, maven, gradle
+ managers: [pip]
+
+ license_compliance_audit:
+ # Required SPDX expression for all source files.
+ required_spdx_expression: "Apache-2.0"
+ # Source paths to audit (relative to upstream repo root).
+ source_paths: [src/, lib/]
+ # Paths to skip (test fixtures, vendored code, etc.).
+ skip_paths: [tests/fixtures/, vendor/]
+
+ flaky_test_triage:
+ # Audit window in days.
+ window_days: 30
+ # Minimum failure rate (fraction) to flag a test as candidate flaky.
+ failure_rate_threshold: 0.1
+```
+
+The config file will land in a separate build item once at least one
+candidate skill reaches the planning stage.
+
+---
+
+## Cross-references
+
+- [`docs/modes.md` § Triage](../modes.md#triage) — mode taxonomy; repo-health
+ skills are listed in the Triage table.
+-
[`tools/spec-loop/specs/repo-health-family.md`](../../tools/spec-loop/specs/repo-health-family.md)
+ — functional spec: acceptance criteria, validation commands, and known gaps.
+-
[`tools/spec-loop/specs/triage-mode.md`](../../tools/spec-loop/specs/triage-mode.md)
+ — parent spec that identified the family gap.
diff --git a/tools/spec-loop/specs/repo-health-family.md
b/tools/spec-loop/specs/repo-health-family.md
new file mode 100644
index 00000000..da8c9046
--- /dev/null
+++ b/tools/spec-loop/specs/repo-health-family.md
@@ -0,0 +1,116 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+---
+title: Repo-health audits
+status: experimental
+kind: feature
+mode: Triage
+source: >
+ MISSION.md § Technical scope (Triage). triage-mode.md Known Gaps
+ ("Repo-health audits are a one-off with no family around them;
+ ci-runner-audit is a standalone read-only audit"). Family design in
+ docs/repo-health/README.md. ci-runner-audit is the first shipped member.
+acceptance:
+ - Every repo-health skill is read-only or proposes-then-confirms; none
+ modifies workflow files, runner configs, or dependency files without
+ human confirmation.
+ - ci-runner-audit validates under skill-and-tool-validate and ships an
+ eval suite.
+ - docs/modes.md Triage table lists each shipped repo-health skill.
+---
+
+# Repo-health audits
+
+## What it does
+
+Surface repository-level health signals a maintainer would otherwise detect
+by hand: obsolete runner labels, Actions workflow security vulnerabilities,
+stale or vulnerable dependencies, license/NOTICE compliance drift, and
+flaky-test patterns. Every skill in the family produces a structured report
+and proposes remedies for human review; no skill applies a change without
+explicit maintainer confirmation.
+
+The family extends the Triage mode's classify-and-propose discipline to
+repository maintenance. The canonical sequence is: fetch a bounded snapshot
+of the target artefacts → classify each against a rule set → present a
+grouped, prioritised report → wait for confirmation before any write.
+
+## Where it lives
+
+- Skill: `ci-runner-audit` — first shipped member. Reads GitHub Actions
+ workflow files across one repo, an explicit set, one Apache project's
+ repos, or the full Apache GitHub org and surfaces obsolete
+ GitHub-hosted runner labels and macOS architecture mismatches. Read-only;
+ no workflow files are modified. Ships `mode: Triage` + `experimental`.
+- Design docs: `docs/repo-health/README.md` — family overview, candidate
+ skills (workflow-security-audit, dependency-audit, license-compliance-audit,
+ flaky-test-triage), and the planned adopter-config scaffold.
+- Planned adopter config: `projects/_template/repo-health-config.md` —
+ per-skill switches (deprecated-runner families, zizmor rule classes,
+ dependency-manager selection, SPDX expression, flaky-test window).
+ Lands in a separate build item once at least one candidate skill reaches
+ the planning stage.
+
+## Behaviour & contract
+
+- **Read-only by default.** The core loop of every repo-health skill is
+ read → classify → present. State changes (filing a tracking issue,
+ opening an update PR) are explicit proposals the maintainer confirms;
+ no skill performs an unconfirmed write to the upstream repo, the
+ tracker, or any config file.
+- **Point-in-time snapshot.** Each run produces a bounded audit report —
+ not a continuous monitor. Re-runs extend or replace the snapshot;
+ no incremental state is persisted between runs.
+- **Authenticated read only.** Skills use `gh api` (read-scoped) for
+ private repos; no write OAuth scopes are requested or used.
+- **One skill, one rule family.** Each candidate skill targets a distinct
+ maintenance concern (runner labels, workflow security, dependency
+ freshness, license compliance, test stability). Skills do not share
+ intermediate state, so each runs independently.
+
+## Out of scope
+
+- **Applying changes.** Updating a workflow file, opening a dependency-bump
+ PR, or correcting a license header are maintainer actions. Skills propose;
+ humans act.
+- **Continuous monitoring.** That is a CI / GitHub Actions job responsibility.
+ Each skill run is a triggered, bounded audit.
+- **Auto-merge of housekeeping changes.** Runner-label fixes and NOTICE
+ updates are eligible for Auto-merge only once that mode is on; the
+ repo-health skills never trigger a merge autonomously.
+- **Security-class vulnerabilities.** CVE-rated dependency issues flow
+ through the security-issue lifecycle
+ ([security-issue-lifecycle.md](security-issue-lifecycle.md)); the
+ `dependency-audit` candidate handles non-CVE dependency hygiene.
+
+## Acceptance criteria
+
+1. Every repo-health skill is read-only or proposes-then-confirms; no skill
+ modifies workflow files, dependency manifests, or source files without
+ human confirmation.
+2. `ci-runner-audit` validates under `skill-and-tool-validate` and ships an
+ eval suite under `tools/skill-evals/evals/ci-runner-audit/`.
+3. `docs/modes.md` Triage table carries each shipped repo-health skill.
+
+## Validation
+
+```bash
+test -f .claude/skills/magpie-ci-runner-audit/SKILL.md
+uv run --project tools/skill-and-tool-validator --group dev
skill-and-tool-validate
+```
+
+## Known gaps
+
+- **Family is one skill deep.** `ci-runner-audit` is the sole shipped
+ member. Four candidates are designed in `docs/repo-health/README.md`:
+ `workflow-security-audit` (zizmor-backed, highest priority — the tool
+ is already in pre-commit), `dependency-audit`, `license-compliance-audit`,
+ and `flaky-test-triage`. Each is a separate build item, sequenced after
+ `ci-runner-audit`'s pilot evaluation confirms the family shape.
+- **No adopter-config scaffold yet.**
`projects/_template/repo-health-config.md`
+ is planned alongside the first candidate skill; the keys are sketched in
+ `docs/repo-health/README.md § Adopter contract`.
+- **`ci-runner-audit` eval suite exists but no adopter pilot has run.**
+ The skill is `experimental`; behaviour may change as cross-org run
+ volumes expose edge cases in runner-label classification.
