SEZ9 opened a new pull request, #12538:
URL: https://github.com/apache/seatunnel/pull/12538
### Purpose of this pull request
Closes #12534. Part of #11616.
Two problems made the accuracy benchmark impractical to run while iterating,
which in turn means accuracy changes tend to land without a measured
before/after.
**1. There was no fast subset.** 100 tasks is the right size for a published
comparison and far too slow for a per-change check, so in practice it got
skipped. `--tasks` exists, but a hand-typed id list differs from run to run, so
the numbers are not comparable — which defeats the purpose.
This adds named, version-controlled subsets in `benchmark/presets.json`,
selected with `--preset`. Because the selection is checked in, repeated runs
stay comparable. `--preset` and `--tasks` are mutually exclusive, and a preset
that cannot resolve every one of its ids under the chosen `--tiers`/`--suite`
fails rather than silently running a smaller set:
```
$ python -m benchmark.runner --preset smoke --tiers 1
error: Preset smoke selects tasks outside --tiers [1] or --suite baseline:
t2_cdc_pg_kafka, t2_mt_table_list, ...
```
One preset ships, `smoke`: 12 tasks, four per tier, ten categories, three
Chinese prompts, deliberately including the four task shapes most models get
wrong (Doris option rules, PostgreSQL CDC prerequisites, conditional routing,
wide DAG wiring), so a regression in prompt or connector-metadata handling
surfaces there first.
**2. `--level l1` was not actually infrastructure-free.** `--level l1` means
static scoring only — no engine, no Docker — but `resolve_levels()` probed for
an engine backend unconditionally, and the probe raised instead of reporting a
missing capability when a `docker` binary is present but not executable by the
current user, which is the normal situation when the user is not in the
`docker` group:
```
PermissionError: [Errno 13] Permission denied: 'docker'
File "benchmark/execution.py", line 131, in docker_engine_available
```
`docker_engine_available()` caught `FileNotFoundError` but not
`PermissionError`, even though its whole contract is to report a capability
rather than raise. Static runs now skip the probe entirely, and the probe
returns `False` on any `OSError`.
### Does this PR introduce _any_ user-facing change?
Yes, in the unreleased `dev` branch only, and additive: a new `--preset`
option for the benchmark runner, documented in `benchmark/README.md`. `--level
l1` now works on a machine with no Docker access instead of raising
`PermissionError`. Existing `--tasks`, `--tiers` and `--suite` behaviour is
unchanged.
### How was this patch tested?
Preset expansion and all four rejection paths were exercised directly: 12/12
ids resolve to 4/4/4 across tiers with 10 distinct categories; narrowing
`--tiers` fails loudly; `--preset` with `--tasks` is rejected by argparse; an
unknown preset name lists the available ones. `resolve_levels("l1", ...)` no
longer calls the probe, and `docker_engine_available()` returns `False` rather
than raising where `docker` is not executable.
The lane itself, `claude-haiku-4.5` via Bedrock, `--level l1`:
```
$ python -m benchmark.runner --preset smoke --level l1 \
--provider bedrock --model us.anthropic.claude-haiku-4-5-20251001-v1:0
```
**9.9 minutes for 12 tasks** (593s total, 49.4s/task), against 100 tasks for
the full suite — short enough to run on every change, which was the point.
Result, with #12536 also applied (that PR fixes two validation artifacts
this lane exposed):
```
pass@1: 66.7% tier1 50.0% tier2 75.0% tier3 75.0%
```
Reported as a reference point rather than a model comparison: one model, one
trial, static gate only. What matters is that it is now cheap to produce the
same 12 numbers before and after a change.
Worth recording for whoever picks up the accuracy work — the lane
immediately surfaced that **3 of the 12 failures are `generation_error:
non-config result: chat`**, i.e. the planner classifying an explicit
pipeline-creation request as conversation. It is non-deterministic (a different
set of tasks each run, and `t1_fake_console` passes when run alone), and at 25%
of the lane it is a larger cluster than any config-quality issue in it. Filed
separately rather than fixed here.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
