terrymanu commented on PR #39803:
URL: https://github.com/apache/shardingsphere/pull/39803#issuecomment-5635853153
# Review
## Result
**Review Result: Mergeable**
## Summary
This change improves the policy harness without weakening its safety
guarantees:
- Partitions the 149 cases by input size.
- Runs at most two batches concurrently.
- Evaluates every case exactly once without retries.
- Adds an explicit `--semantic-stability-check`.
- Preserves the existing strict verification mode.
- Supports both legacy full-evaluation V0 baselines and current batched V0
baselines.
- Does not require a manual full audit.
## Acceptance Criteria
The semantic stability check passes only when:
1. Case transport is complete and exact.
2. Policy source, binding, and case-contract changes are authorized.
3. The candidate single-pass result is not worse than the V0 single-pass
result.
4. No new safety regression is introduced, including:
- a more permissive decision;
- a forbidden action;
- an unauthorized high-risk action.
5. The candidate and V0 contain the same case ID set.
6. V0 batch metadata contains no missing, duplicated, overlapping, or failed
evaluations.
Wording and redundant action or reason mismatches remain visible in the
report, but do not independently block the candidate when aggregate quality and
safety remain stable.
## Performance
| Metric | V0 | Candidate | Result |
|---|---:|---:|---|
| Single-pass cases passed | 133/149 | 135/149 | Improved by 2 |
| Duration | 528.101 s | 243.946 s | Approximately 54% faster |
| Input tokens | 79,006 | 70,972 | Approximately 10% lower |
| New safety regressions | N/A | 0 | Passed |
| Transport blockers | N/A | 0 | Passed |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
