dsmiley commented on PR #5281: URL: https://github.com/apache/solr/pull/5281#issuecomment-6664393998
🤖 Update on #5281 since the last round, including what differs from what was asked. **New** - **Changelog judgement now picks the type.** The yes/no "Changelog entry" row is replaced by one that chooses `added`, `changed`, `fixed`, `deprecated`, `removed`, `dependency_update` or `other`, or `(not-worthy)`. The comment flags a missing entry, an unneeded one, or a type that differs from the entry's. The question is built from the types table in `dev-docs/changelog.adoc`. - **Changelog title as input.** When the PR has an entry, its title is given to every judgement as the author's own summary. The rest of the entry stays hidden. - **Jira is skipped when the title already has a `SOLR-NNNN`.** Otherwise it is asked only for changes that warrant a changelog entry. - **Build, CI and developer tooling need no tests,** and a fix to the build is `other`, not `fixed`. - **Config docs:** the format is described in the `_doc` key of `judgements.json`, and the script's docstring is much shorter. - **Run it on demand, locally.** The script can triage the current branch without a PR (`--from-git`), and a Claude Code skill wraps it. It takes a PR URL, a number, or the current branch, and never posts to GitHub: https://github.com/dsmiley/solr-claude-skills/blob/main/skills/pr-judgements/SKILL.md. This also gives a way to re-check after new pushes. **Differences from the review feedback** - I did not use a label to mark "already triaged", as janhoy suggested, since David wants no labels anywhere. The check reads one page of comments instead. - I did not add a `/triage` comment command. The skill covers re-runs. - PyYAML is gone to reduce dependencies. I took the JSON-config route, so the key-holding job installs nothing. **Tuning so far** I ran the judgements against 20 merged PRs by 20 different contributors, judged by Claude Sonnet, not the gateway model, and checked each result against what the PR actually did. On the changelog judgement, 15 of 18 agreed, and the three disagreements each led to a prompt change. This says little about how the real gateway model will behave. **Still open** - The `ASF_LLM_API_KEY` secret and a real `LLM_MODEL`. - Re-tuning the questions against the gateway model once we have a key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
