SEZ9 opened a new issue, #12534: URL: https://github.com/apache/seatunnel/issues/12534
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description Two problems make the `seatunnel-cli` accuracy benchmark impractical to run while iterating, which in turn means accuracy changes tend to land without a measured before/after. **1. There is no fast subset.** The suite is 100 tasks across three tiers. That is the right size for a published comparison but far too slow to run on every change, so in practice it gets skipped. `--tasks` exists, but a hand-typed id list differs from run to run, so the resulting numbers are not comparable — which defeats the purpose. **2. `--level l1` is not actually infrastructure-free.** `--level l1` means static scoring only: no engine, no Docker. But `resolve_levels()` probes for an engine backend unconditionally, and that probe crashes rather than reporting a missing capability when a `docker` binary is present but not executable by the current user — the normal situation when the user is not in the `docker` group: ``` PermissionError: [Errno 13] Permission denied: 'docker' File "benchmark/execution.py", line 131, in docker_engine_available ``` `docker_engine_available()` catches `FileNotFoundError` but not `PermissionError`, even though its contract is to report a capability rather than raise. ### Proposal - Add named, version-controlled task subsets in `benchmark/presets.json`, selected with `--preset`. Because the selection is checked in, repeated runs stay comparable. Make `--preset` and `--tasks` mutually exclusive, and fail rather than silently running a smaller set when a preset cannot resolve all of its ids under the chosen `--tiers`/`--suite`. - Ship one preset, `smoke`: 12 tasks, four per tier, ten categories, three Chinese prompts, deliberately including the four task shapes most models get wrong (Doris option rules, PostgreSQL CDC prerequisites, conditional routing, wide DAG wiring) so a regression in prompt or connector-metadata handling surfaces there first. - Skip the engine probe entirely for `--level l1`, and make `docker_engine_available()` return `False` on any `OSError`. Related umbrella: #11616. ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
