Hi Dave,

I tested it by running it blind over 50 release candidates from 40 podlings, 
voted between January 2024 and September 2026. For each one, I read only the 
opening vote email, wrote down the findings, then read the thread and compared. 
It made 46 correct calls and one partial. It missed a real blocker twice and 
raised one false blocker. It also found 13 problems no reviewer mentioned, so 
it could be argued from those numbers that it does better than human review on 
its own. Compared to the plain skill, which only got 39 of the 50 right.

The method and the list of podlings are in the README, and anyone can repeat 
it. The Python checker, check_rc.py, is deterministic and needs no LLM, so that 
is useful on its own.

On the roster point, the onboarding skill doesn't depend on the Whimsy roster 
tool. If the roster tool is replaced, only the MCP underneath might need to 
change, not the skill.

For the minimum LLM, only reviewing what the checker flags needs a model, and a 
human could also review it, so no model would be needed. The SKILL does make a 
few other checks by calling MCPs, but these are optional. I have not fully 
tested it without the MCPs to see what difference they make, but it would make 
a difference; for instance, it wouldn’t look at previous RCs/votes to see what 
issues they contained. I've run the skill with Claude, Codex, and OpenCode. The 
results with OpenCode (Zen) were not as good, but still helpful.

Here is what the checker output looks like before it gets parsed by an LLM:

# Release candidate check: 0.43.0-incubating-RC1

Deterministic checks only. `fail` items are facts. `candidate` items need a 
person;
most will be fine and some will not. Nothing here is a vote.

## Artifacts
- apache-burr-0.43.0-incubating-sdist.tar.gz
- apache-burr-0.43.0-incubating-src.tar.gz
- apache_burr-0.43.0-py3-none-any.whl

## Counts
- files: 1152
- asf_header: 869
- no_header: 17
- spdx: 0
- third_party_header: 1
- asf_plus_foreign: 0
- binary: 0
- nested_archive: 0
- media: 90
- data: 0
- licence_signal: 1

## fail (1)
### naming: archive name does not contain "incubating" (1)
- `apache_burr-0.43.0-py3-none-any.whl`

## candidate (2)
### licence-signal: mentions GPL/LGPL/AGPL (1)
- `apache-burr-0.43.0-incubating-src.tar.gz/website/package-lock.json` | 
2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==",       "cpu": [    
     "arm64"       ],       "license": "LGPL-3.0-or-later",       "optional": 
true,       "os": [         "darwin"       ],       "funding": {         "url": 
"https://o
### header-third-party: third-party header (Apache-licensed or other 
copyright); check LICENSE and NOTICE cover it (1)
By copyright holder: shadcn (1)
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/button.tsx` 
| ["* Copyright (c) 2023 shadcn"]

## Images, fonts and documents (90)
Usually fine. Fonts and icon sets are the ones that tend to carry their own 
licence.
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/burr/tracking/server/s3/architecture.png`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/hello-world-counter/statemachine.png`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/statemachine.png`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/streaming-fastapi/statemachine.png`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/multi-modal-chatbot/statemachine.png`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/email-assistant/statemachine.png`
- `apache-burr-0.43.0-incubating-sdist.tar.gz/telemetry/ui/public/favicon.ico`
- `apache-burr-0.43.0-incubating-sdist.tar.gz/telemetry/ui/public/logo.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/chatbot.gif`
- `apache-burr-0.43.0-incubating-src.tar.gz/burr_logo.svg`
- `apache-burr-0.43.0-incubating-src.tar.gz/burr_logo.png`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/burr/tracking/server/s3/architecture.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/burr_logo.svg`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/burr_logo.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/logos/taskhuman.svg`
- ... 75 more in findings.json

## Code or documentation files with no licence header (17)
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/prompts.py`
- 
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/deep_researcher_utils.py`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/safari.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/animated-beam.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/shimmer-button.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/icon-cloud.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/border-beam.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/dot-pattern.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/marquee.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/number-ticker.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/magic-card.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/animated-shiny-text.tsx`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/blur-fade.tsx`
- `apache-burr-0.43.0-incubating-src.tar.gz/examples/deep-researcher/prompts.py`
- 
`apache-burr-0.43.0-incubating-src.tar.gz/examples/deep-researcher/deep_researcher_utils.py`
- ... 2 more in findings.json

Kind regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to