janhoy opened a new pull request, #887: URL: https://github.com/apache/solr-operator/pull/887
🤖 AI assisted 🤖 Adds an advisory, LLM-assisted triage comment on new pull requests, modelled on the workflow Solr recently adopted in apache/solr ([`pr-triage-llm.yml`](https://github.com/apache/solr/blob/main/.github/workflows/pr-triage-llm.yml) and [`judgements.json`](https://github.com/apache/solr/blob/main/.github/pr-triage/judgements.json)). Once per PR (opened, or draft marked ready), a model behind the ASF LLM gateway (llm.apache.org) is asked what the change *warrants*, and the answers are compared with what the PR *includes*. One comment is posted, highlighting mismatches; when every judgement agrees it is a single green line instead of a table (a deliberate difference from Solr). It never fails the check, posts nothing if the model gives no usable answer, and does not re-run on later pushes. Changelog, docs, README and test files are hidden from the model, so it judges the change on its merits rather than noticing what the author added. ## What it checks The judgements in `.github/pr-triage/judgements.json` encode the feedback contributor PRs to this repo have historically received from reviewers and that CI does not catch. I went through the review threads of ~45 non-committer PRs; the recurring asks were, in order: a changelog entry in `helm/solr-operator/Chart.yaml` (by far the most common, including for controller-only changes), a row in the chart README values table, a new CRD field not wired through the controllers or not exposed in the `solr` chart, tests, Solr 8.11 vs 9.x compatibility, and a note on how the change was tested. Nobody asked for issue links, DCO or squashing, so unlike Solr there is no Jira-style judgement. | Judgement | Compared with | |---|---| | Changelog entry and its `kind` | `artifacthub.io/changes` in `helm/solr-operator/Chart.yaml` | | Same for the `solr` chart | `helm/solr/Chart.yaml` (only asked when that chart changed) | | Helm README values row | `helm/*/README.md` (only when values/templates changed) | | User docs | `docs/modules/**` | | Upgrade notes (removed fields, changed defaults, pod spec changes, new RBAC, raised minimum versions) | `docs/modules/upgrade-notes/**` | | Tests | `*_test.go`, `tests/e2e/**` | | New CRD option complete (wired through, markers, exposed in `solr` chart) | flag: a `true` verdict is the finding | | Depends on a Solr/Kubernetes version newer than the oldest supported | flag | | Cluster-dependent change without a description of how it was tested | flag | ## Files - `.github/workflows/pr-triage-llm.yaml`: Solr's two-job design. `pull_request_target` so fork PRs can use the secret; the job holding the key has a read-only token and the job that comments never sees the key. No PR code is checked out or run. Skips quietly when the secret is absent, so forks are unaffected. - `hack/pr-triage/pr_triage_llm.py`: Solr's script with four generic additions: project context for the system prompt, `applies_to` (a judgement is only asked when certain files changed), `evidence_text` (regex on the PR body), and flag judgements with no evidence file. - `hack/pr-triage/test_pr_triage_llm.py`: 55 offline tests, ported from Solr and extended; wired into `check.yaml`. - `dev-docs/pr-triage-llm.md`: how it works, local use, setup. Linked from the README. ## Trying it on an existing PR Only Python 3, `gh` and `jq` are needed, plus a personal key for llm.apache.org. From this branch: ```bash export LLM_API_KEY='<your llm.apache.org token>' export LLM_MODEL=qwen3.8-27b triage_pr() { local n=$1 repo=apache/solr-operator dir=/tmp/triage-$1 mkdir -p "$dir" gh api "repos/$repo/pulls/$n" \ --jq '{title, body, head, base, maintainer_can_modify, changed_files}' > "$dir/pr.json" gh api --paginate "repos/$repo/pulls/$n/files" \ --jq '.[] | {filename, status, additions, deletions, patch}' | jq -s . > "$dir/files.json" python3 hack/pr-triage/pr_triage_llm.py --config .github/pr-triage/judgements.json \ --pr "$dir/pr.json" --files "$dir/files.json" --out "$dir/result.json" "${@:2}" > /dev/null [ -f "$dir/result.json" ] && jq -r .comment "$dir/result.json" } triage_pr 880 # the comment as it would be posted triage_pr 571 --dry-run # just the prompt, no model call ``` The Why column is HTML-entity escaped on purpose (model text cannot inject links or mentions); GitHub renders it normally. Add `| python3 -c 'import html,sys; print(html.unescape(sys.stdin.read()))'` to read it in a terminal. Run on #880 it agreed on all eight evidence-backed judgements and raised one flag: the feature needs Solr 10 and the version gate is a TODO, which the PR's own comments confirm. The offline tests run with `python3 -m unittest discover -s hack/pr-triage -p 'test_*.py'`. ## Before this is live - Repository secret `ASF_LLM_API_KEY` (an automation key from llm.apache.org, via ASF Infra). Optional variable `LLM_MODEL`. - Expect to tune the `question` wording in `judgements.json` as real comments come in; that is the only knob that usually matters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
