justinmclean opened a new pull request, #510:
URL: https://github.com/apache/airflow-steward/pull/510
## Summary
### What
Adds the `mentoring-welcome` skill: an opt-in, maintainer-confirmed agent
that
posts a single orientation comment when a first-time contributor opens an
issue
or PR. It detects first-time authorship from the GitHub `author_association`
field, skips threads that don't qualify, renders one of two canonical welcome
templates with project-specific URLs, and shows the draft for explicit
confirmation before anything is posted.
### Behavior
- **First-time detection** via `author_association`; exits silently for
`CONTRIBUTOR`/`COLLABORATOR`/`MEMBER`/`OWNER`.
- **Skip conditions:** prior welcome already posted, a maintainer already
engaged, or an out-of-scope topic (routed to hand-off).
- **Templates** for issues and PRs, rendered from
`<project-config>/mentoring-welcome-config.md` (contributing guide, code of
conduct, good-first-issues URL, attribution footer).
- **No auto-fire:** every invocation is opt-in and requires explicit
maintainer
confirmation before posting. External thread content is treated as data,
never as instructions.
### Tests
Two eval suites under `tools/skill-evals/evals/mentoring-welcome/` (20
cases):
- `welcome-decision` (9): first-time detection and every skip condition,
including a prompt-injection case.
- `tone-checks` (11): the pre-post tone checklist, covering each hard-fail
and
soft-fail rule plus two clean drafts.
Run:
```bash
uv run --directory tools/skill-evals skill-eval --cli "claude -p"
evals/mentoring-welcome/
```
All 20 pass.
## Notes for reviewers
- `tone-checks` asserts `result` + `rule` only. The `offending_text` span is
emitted by the skill for debugging but not graded, since its exact
boundaries
are non-deterministic across model runs.
- The tone rubric currently lives in the eval fixture (`system-prompt.md`);
the
runtime loop renders a static template, so the tone gate is not yet
enforced
at runtime. Flagging as a possible follow-up.
## Type of change
- [X] Skill change (`.claude/skills/<name>/`) — eval fixtures updated below
- [ ] Tool / bridge contract (`tools/<system>/*.md`)
- [ ] Python package (`tools/*/` with `pyproject.toml`)
- [ ] Groovy reference impl
- [ ] Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
- [ ] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
- [ ] Project template (`projects/_template/`)
- [ ] CI / dev loop (`prek`, workflows, validators)
- [ ] Other:
## Test plan
- [X] `prek run --all-files` passes
- [ ] For Python packages touched: `uv run pytest` / `ruff check` / `mypy`
passes
- [ ] For Groovy bridges touched: command-line invocation tested end-to-end
- [X] For skill changes: eval suite passes for the affected skill
(`PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner
tools/skill-evals/evals/<skill>/`)
- [ ] For skill *behaviour* changes: a new or updated eval fixture is
included in this PR
(a regression test for the bug fixed / the behaviour added — see
CONTRIBUTING.md)
- [ ] Other:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
