goutamadwant opened a new pull request, #12192:
URL: https://github.com/apache/seatunnel/pull/12192
### Purpose of this pull request
Closes #12189.
Related to #11616.
Add an offline comparison for two saved AI CLI benchmark runs. An overall
pass-rate
gain can hide tasks that regressed; this report pairs model/task/trial
identities and
shows first-attempt and repair-budget transitions alongside the aggregate
delta.
### Does this PR introduce _any_ user-facing change?
Yes. From `seatunnel-cli/`:
```bash
python -m benchmark.compare baseline/results.json candidate/results.json
--out comparison.md
```
The command uses the standard library and does not call providers or execute
jobs.
It checks recorded model settings, task fingerprints, requested gates, trial
counts,
and repair budgets. Missing, incompatible, or incomplete evidence is listed
and
excluded from both denominators. Existing output files are never overwritten.
New runs record an additive task fingerprint. Existing single-run reports are
unchanged; older results without fingerprints require fresh runs for
comparison.
The report describes observed changes, not statistical significance or a CI
acceptance
decision. EN/ZH documentation explains compatibility and environment
limitations.
### How was this patch tested?
Added offline acceptance tests for gains hiding regressions, unchanged runs,
missing
and skipped gates, incompatible settings, task changes, duplicate identities,
generation failures, deterministic output, and input/output preservation.
Tests also
exercise the real runner's saved-result path and verify byte-for-byte
unchanged
single-run Markdown and CSV reports.
```bash
cd seatunnel-cli
python -m pytest -q
black --check benchmark/compare.py tests/test_benchmark_comparison.py
ruff check benchmark/compare.py tests/test_benchmark_comparison.py
```
The complete CLI suite passed on Python 3.10 and 3.11: 129 tests and 3
subtests
per runtime, including 46 comparison cases. Black, Ruff, and the root
Spotless
formatter passed. Full-repository `./mvnw -q -DskipTests verify` passed on
Java 11;
that build check skips tests and is separate from the Python suite above.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— No new dependencies or JARs.
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs — EN/ZH updated.
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. — Existing commands and reports are
unchanged.
* [x] If you are contributing the connector code, please check that the
following files are updated: — Not a connector change.
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
