SEZ9 opened a new issue, #11549: URL: https://github.com/apache/seatunnel/issues/11549
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issues. ### Description While working on the SeaTunnel AI CLI (#10789, #11109), we built an accuracy benchmark to answer a question the project currently has no way to answer quantitatively: **when the CLI's prompts, connector metadata, or repair logic change, does config-generation accuracy actually improve — and by how much, for which models?** We would like to discuss whether the community wants this benchmark to live in the main repository (e.g. under `seatunnel-cli/benchmark/`), before we open a PR. ### What it is A standalone, deterministic evaluation harness for the AI CLI (pure additions — no product code changes): - **100 tasks** in three complexity tiers (20 simple / 45 medium / 35 complex-DAG), covering 12 ETL scenario categories (batch sync, file ETL, streaming ingest, CDC, multi-pipeline, conditional routing, ...), with 10 Chinese-language prompts and 18 "rule probes" targeting known LLM failure modes (conditional option misuse, BATCH-vs-STREAMING inference, plugin_input/plugin_output wiring). - **Three verdict gates** mirroring the CLI's own check → dry-run → run pipeline: - L1 static (HOCON parse + connector metadata + assertions, offline); - L2 engine dry-run (`--dry-run static`, per #10763); - L3 real execution — the generated config runs on the official `apache/seatunnel` Docker image against a docker-compose data environment (MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch/MinIO, optional Doris/StarRocks), batch verdict by exit code, streaming by 60s health. - **Repair-loop measurement**: on failure, the failing gate's real error output is fed to the CLI's own repair agent (max 3 rounds), producing pass@1 / pass@≤3 and per-gate repair conversion rates. - All verdicts are deterministic (exit codes, liveness, regex assertions) — no LLM-as-judge. Reports are stamped with the CLI version + git commit, so any CLI change can be A/B measured. Credentials are env-vars only (ASF compliant); results directories are git-ignored. ### What it found (evidence it earns its place) A full run across 7 Bedrock models (Claude Fable 5 / Opus 4.8 / Sonnet 5, GPT-5.6 Terra / Sol, Qwen3-Coder-Next, DeepSeek V3.2) produced actionable results: - **Static rankings invert under real execution**: the static-gate leader dropped to last among the top-3 when configs actually ran (93% static → 74% real), while the static #3 won at 85% — "looks correct" and "actually runs" are different model capabilities, and only L3 distinguishes them. - **13 tasks failed for every model** — clustering into four concrete CLI improvement targets (Doris/StarRocks option knowledge, PostgreSQL-CDC prerequisites, conditional-routing golden examples, complex DAG wiring), i.e. a data-driven roadmap for prompt/metadata work. - The benchmark run itself surfaced four real defects, three now fixed in #11509 and one each reported in #11510 (opengauss-jdbc hijacking the PostgreSQL driver in the official image) and #11511 (released `--check` is a no-op stub). ### Discussion points 1. **In-repo or separate?** Options: (a) `seatunnel-cli/benchmark/` in the main repo (versioned with the CLI it measures — our preference, since the CLI-version stamp makes each PR's accuracy impact measurable); (b) the `apache/seatunnel-tools` repo; (c) keep it external. 2. **Scope of CI integration** (optional, later): a 10–15 task L1-only smoke subset could gate CLI PRs at zero API cost; the full suite needs LLM credentials and is only meaningful as a manually-triggered / release-time job. 3. **Maintenance expectations**: tasks are declarative JSON; adding a scenario is a JSON entry, no harness changes. Docker environment pins the same image versions the e2e suite uses. If the direction is agreed, we will open a PR with the harness + tasks + docs (roughly +4,000 lines, all new files), followed by a docs PR adding a model-selection guide based on the results. ### Usage Scenario CLI contributors measuring accuracy impact of prompt/metadata/repair changes; users choosing an LLM for the CLI based on measured (not anecdotal) accuracy. ### Related issues #10789 (AI CLI), #10763 (dry-run), #11508/#11509 (fixes found by this benchmark), #11510, #11511, #11547 ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
