This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 2c4f51d387 [Docs][CLI] Add AI CLI documentation section (overview,
quickstart, design, model benchmark) (#11551)
2c4f51d387 is described below
commit 2c4f51d387f699ec3418861dd6c18a2c0bd6163f
Author: SEZ <[email protected]>
AuthorDate: Sat Jul 25 16:27:54 2026 +0800
[Docs][CLI] Add AI CLI documentation section (overview, quickstart, design,
model benchmark) (#11551)
Co-authored-by: SEZ9 <[email protected]>
---
docs/en/ai-cli/benchmark.md | 79 ++++++++++++++++++++++++++++++
docs/en/ai-cli/design.md | 89 +++++++++++++++++++++++++++++++++
docs/en/ai-cli/overview.md | 40 +++++++++++++++
docs/en/ai-cli/quickstart.md | 114 +++++++++++++++++++++++++++++++++++++++++++
docs/sidebars.js | 10 ++++
docs/zh/ai-cli/benchmark.md | 79 ++++++++++++++++++++++++++++++
docs/zh/ai-cli/design.md | 89 +++++++++++++++++++++++++++++++++
docs/zh/ai-cli/overview.md | 40 +++++++++++++++
docs/zh/ai-cli/quickstart.md | 114 +++++++++++++++++++++++++++++++++++++++++++
9 files changed, 654 insertions(+)
diff --git a/docs/en/ai-cli/benchmark.md b/docs/en/ai-cli/benchmark.md
new file mode 100644
index 0000000000..f8a20fb11b
--- /dev/null
+++ b/docs/en/ai-cli/benchmark.md
@@ -0,0 +1,79 @@
+---
+sidebar_position: 4
+---
+
+# Model Benchmark
+
+The AI CLI's accuracy is measured — not assumed — by a dedicated benchmark:
100 tasks in three complexity tiers, judged through layered verdict gates up to
**real job execution** against Dockerized data sources, across 7 mainstream
LLMs. This page summarizes the methodology, the results, and what they mean for
choosing a model.
+
+> Results below were measured in July 2026 against seatunnel-cli v0.1.0
(commit `59ada4ec0`) with models served by AWS Bedrock. Accuracy drifts as
models and the CLI evolve; treat the numbers as a snapshot and re-run the
benchmark for current values.
+
+## Methodology
+
+**Task suite**: 100 tasks — 20 simple (single source→sink), 45 medium (type
mapping, CDC, transform chains, multi-table), 35 complex (multi-source DAGs,
fan-out, conditional routing) — across 12 ETL scenario categories, with 10
Chinese-language prompts and 18 rule probes targeting known LLM failure modes
(conditional option misuse, BATCH/STREAMING inference, routing-label wiring).
+
+**Verdict gates**, mirroring the CLI's own check → dry-run → run pipeline:
+
+| Gate | Verdict | Catches |
+|------|---------|---------|
+| L1 static | HOCON parse + connector metadata + assertions | Syntax, wrong
connector, missing options |
+| L3 real execution | Job runs on the official `apache/seatunnel` image
against real MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch; batch = exit
code, streaming = 60s health | Everything "looks right but doesn't run" |
+
+**Repair loop**: on failure, the failing gate's real error output is fed to
the CLI's own repair agent, up to 3 rounds. All verdicts are deterministic — no
LLM-as-judge.
+
+## Headline Result: Static Rankings Invert Under Real Execution
+
+| Model | Static gate (L1) | Real execution (L1+L3) |
+|-------|:---:|:---:|
+| Claude Opus 4.8 | 89% (rank 3) | **85% (rank 1)** |
+| GPT-5.6 Sol | 90% (rank 2) | 81% (rank 2) |
+| GPT-5.6 Terra | **93% (rank 1)** | 74% (rank 3) |
+
+The static leader wrote configs that *looked* correct but failed at runtime
(−20pp static→real decay); the static #3 wrote configs that ran (−6pp decay).
**"Looks correct" and "actually runs" are different model capabilities** —
which is why this benchmark executes configs for real, and why accuracy claims
based on static checks alone should be treated with caution.
+
+## Full Static-Gate Ranking (7 models)
+
+| Model | Pass (≤5 repairs) | First-try pass | Repaired |
+|-------|:---:|:---:|:---:|
+| GPT-5.6 Terra | 93% | 79% | +14 |
+| GPT-5.6 Sol | 90% | 80% | +10 |
+| Claude Opus 4.8 | 89% | **87%** | +2 |
+| Claude Sonnet 5 | 82% | 78% | +4 |
+| Claude Fable 5 | 80% | 73% | +7 |
+| Qwen3-Coder-Next | 67% | 53% | +14 |
+| DeepSeek V3.2 | 58% | 58% | +0 |
+
+## Model Behavior Profiles
+
+Generation ability and repair ability are **independent dimensions** — and
family traits are consistent:
+
+- **First-try-correct (Anthropic family)**: Opus 4.8 has the highest first-try
rate (87% static / 77% real) and the smallest static→real decay — what it
writes tends to run. But its repair contribution is minimal (+2): it rarely
recovers from its own failures.
+- **Iterative-repair (GPT-5.6 Terra)**: unremarkable first-try, but the
strongest repairer — it fixed 48% of its real runtime failures, climbing from
55% to 74%. Its drafts carry the most "looks-right-but-won't-run" risk.
+- **Repair-disabled (DeepSeek V3.2)**: zero successful repairs in 100 tasks —
its repair attempts return the original config or give up. With a product built
around an auto-repair loop, this effectively halves the product's mechanism.
+- Notable: the deep-reasoning flagship (Fable 5) ranked only 5th — it
over-asks clarifying questions where the task calls for sensible defaults. And
on the 10 Chinese-language tasks the Chinese-developed models scored *lower*
(Qwen 5/10, DeepSeek 4/10) than GPT-5.6 Sol (10/10).
+
+## Model Selection Guide
+
+| Scenario | Recommendation | Why |
+|----------|----------------|-----|
+| Interactive use (user waiting) | GPT-5.6 Sol / Terra | Fast (25–58s/task),
high combined accuracy |
+| Unattended / batch generation | Claude Opus 4.8 | Highest first-try rate;
output most likely to run unmodified |
+| Budget-sensitive regression runs | Qwen3-Coder-Next | Lowest cost tier;
acceptable for smoke testing, not production generation |
+| Avoid for auto-repair workflows | DeepSeek V3.2 | Repair success rate of
zero in measurement |
+
+## Known Weak Scenarios (all models)
+
+Thirteen tasks failed for **every** top-3 model. If your pipeline matches one
of these shapes, review the generated config before production use:
+
+| Scenario | Failure root cause |
+|----------|--------------------|
+| Doris / StarRocks sinks | Connector option knowledge gaps (fenodes, load
ports, save modes) |
+| PostgreSQL-CDC | Missing prerequisites (replication slots, publications) not
reflected in config |
+| Conditional routing (one source split to multiple sinks by predicate) |
`plugin_output`/`plugin_input` wiring with parallel SQL transforms |
+| Wide DAGs (5+ blocks, mixed transforms) | Label pairing and block-ordering
mistakes |
+
+These clusters are engineering targets, not permanent limits: they are being
addressed through connector-metadata injection and golden-example coverage,
with improvements verified against this same benchmark.
+
+## Repair Loop: Measured Effectiveness
+
+Feeding **real engine errors** back to the repair agent recovered 47% of
runtime failures (top-3 models combined). The same model repairs structured
validation errors at ~2× the rate of raw Java stack traces — evidence that
structured error parsing, not a smarter model, is the highest-leverage next
improvement for the repair loop.
diff --git a/docs/en/ai-cli/design.md b/docs/en/ai-cli/design.md
new file mode 100644
index 0000000000..6db975961f
--- /dev/null
+++ b/docs/en/ai-cli/design.md
@@ -0,0 +1,89 @@
+---
+sidebar_position: 3
+---
+
+# Design
+
+This page explains how the AI CLI turns a natural-language request into a
runnable config, and why it is built as a multi-agent pipeline with layered
validation rather than a single LLM call.
+
+## Why Not a Single LLM Call
+
+SeaTunnel config generation is a domain-DSL problem with failure modes that a
plain "prompt in, config out" call cannot handle reliably:
+
+- **150+ connectors, 20–50 options each** — no model has complete, current
knowledge of every option name and type;
+- **conditional option dependencies** — e.g. text-format-only options are
runtime errors when the format is PARQUET;
+- **DAG wiring semantics** — `plugin_output`/`plugin_input` labels must pair
exactly across source/transform/sink blocks;
+- **mode inference** — CDC sources require STREAMING; bounded sources should
not carry streaming-only options.
+
+The CLI therefore grounds the model with authoritative connector metadata,
validates every candidate config, and repairs failures with real error feedback.
+
+## Multi-Agent Pipeline
+
+```
+User input (natural language)
+ │
+ ▼
+┌─────────────────┐ ┌──────────────────────┐
+│ Planner Agent │───▶│ Connector Knowledge │
+│ (intent + │◀───│ Base (tools) │
+│ clarification)│ └──────────────────────┘
+└────────┬────────┘
+ │ structured plan
+ ▼
+┌─────────────────┐
+│ Config Agent │ Generate HOCON config
+└────────┬────────┘
+ ▼
+┌─────────────────┐ ┌──────────────────────┐
+│ Validator Agent │───▶│ Local validation + │
+│ │ │ engine --check │
+└────────┬────────┘ └──────────────────────┘
+ │
+ PASS ── Yes ─▶ Output + auto-save
+ │
+ No (max 3 rounds)
+ ▼
+┌─────────────────┐
+│ Fix Agent │ Correct errors, re-validate
+└─────────────────┘
+
+/check or /run failure at any later point
+ ▼
+┌─────────────────┐
+│ Repair Agent │ Diagnose real engine/runtime errors, patch config
+└─────────────────┘
+```
+
+**Planner** classifies intent (new pipeline vs question vs error diagnosis),
selects connectors, and asks a clarifying question only when no reasonable
default exists. **Config Agent** writes HOCON grounded in per-connector
metadata injected into its prompt. **Validator** combines deterministic local
checks with an LLM semantic review. **Fix/Repair Agents** receive the actual
error text — validation findings during generation, or real engine stack traces
after `/check`//`run` — and patch [...]
+
+## Connector Knowledge Base
+
+Two-tier resolution keeps option knowledge accurate without hand-maintained
prompt text:
+
+1. **Runtime API** — a running SeaTunnel engine's option-rules endpoint
(always current);
+2. **Bundled metadata** — `connector_metadata.json`, exported from the engine
via reflection, shipped with the CLI. Includes required/optional options,
conditional groups, and value constraints for sources, sinks, and transforms.
+
+Prompts are assembled per request: only the metadata for the connectors the
Planner selected is injected, with conditional options explicitly grouped under
their trigger conditions ("only include when `format = text`") — the single
most effective measure against invented or misplaced options.
+
+A three-layer generation strategy stacks on top: **Skill SOPs** (scenario
playbooks — batch sync, CDC realtime, multi-pipeline, ...) → **Golden
Examples** (verified working configs for common pairs) → **Connector Metadata**
(authoritative option rules).
+
+## Validation Pipeline
+
+| Phase | Method | Catches |
+|-------|--------|---------|
+| 1. Local | HOCON syntax, structure, required options vs metadata,
routing-label pairing, unresolved `${VAR}` placeholders (field-aware: engine
template variables like `${now}` in `file_name_expression` are exempt),
security checks | Syntax errors, missing/unknown options, wiring mistakes |
+| 2. Engine dry-run | `seatunnel.sh --check` / `--dry-run static` — the
engine's real parse path | Plugin loadability, option types, unknown keys, DAG
topology |
+| 3. Execution | `/run` via REST API or `seatunnel.sh` | Everything runtime:
connectivity, schemas, CDC prerequisites |
+
+Each layer is a strict superset of certainty; failures at any layer feed the
repair loop with that layer's real error output.
+
+## Design Principles
+
+- **Ground, don't trust**: every connector fact in the prompt comes from
engine metadata, not model memory.
+- **Validate deterministically, repair with the LLM**: verdicts come from
parsers and the engine; the LLM's job is generation and repair, never judging
correctness.
+- **Real errors are the best prompts**: repair agents receive actual
validation output and stack traces. Measured accuracy confirms structured,
specific errors repair far better than raw noise (see
[benchmark](benchmark.md)).
+- **Security by default**: generated configs use `${ENV_VAR}` placeholders for
all credentials; session storage redacts secrets; `/remember` rejects sensitive
values.
+
+## Measured, Not Assumed
+
+The architecture is evaluated by a [dedicated benchmark](benchmark.md) — 100
tasks, three verdict gates including real execution against Dockerized data
sources, across 7 LLMs. Benchmark results directly drive the improvement
roadmap (connector-knowledge injection, structured error parsing for the repair
loop), and every future change to prompts or metadata can be A/B measured
against the same task suite.
diff --git a/docs/en/ai-cli/overview.md b/docs/en/ai-cli/overview.md
new file mode 100644
index 0000000000..bd505651be
--- /dev/null
+++ b/docs/en/ai-cli/overview.md
@@ -0,0 +1,40 @@
+---
+sidebar_position: 1
+---
+
+# AI CLI Overview
+
+The SeaTunnel AI CLI generates production-ready SeaTunnel pipeline
configurations from natural language. Describe a data synchronization task in
English or Chinese, and the CLI produces a validated HOCON config file — with
automatic validation, error repair, and one-click execution.
+
+```
+🐬 SeaTunnel > Sync the users table from MySQL to S3 as Parquet files
+
+ ⚙️ Generating SeaTunnel config...
+ ✅ Validating config (round 1)...
+
+ 📋 Generated SeaTunnel Config
+ Config saved to: .data/last_job.conf
+```
+
+The AI CLI ships as the `seatunnel-cli` module inside the main SeaTunnel
repository and is packaged into the standard distribution. It is launched via
`bin/seatunnel-ai.sh` from a distribution, or installed from source with pip.
+
+## Key Capabilities
+
+- **Natural language to config** — English or Chinese input, complete HOCON
output
+- **Multi-provider LLM** — AWS Bedrock (including OpenAI-family models via the
`bedrock-mantle` endpoint), Anthropic API, OpenAI and compatible APIs
+- **Multi-agent pipeline** — Planner → Config Generator → Validator →
auto-fix, with up to 3 correction rounds
+- **Connector knowledge** — 150+ connectors with full option rules and value
constraints, resolved from a live engine or bundled metadata
+- **Validation & repair** — local checks, engine `--check`/dry-run, and
LLM-powered diagnosis-and-repair when `/check` or `/run` fails
+- **Session & memory** — multi-turn refinement, persistent sessions,
connection-detail memory (credentials are never stored)
+
+## Documentation in this Section
+
+| Page | Contents |
+|------|----------|
+| [Quick Start](quickstart.md) | Install, configure a provider, generate your
first pipeline |
+| [Design](design.md) | The multi-agent architecture and validation pipeline |
+| [Model Benchmark](benchmark.md) | Measured accuracy across 7 LLMs, model
selection guidance, and known weak scenarios |
+
+## Relationship to Other AI Tooling
+
+The AI CLI is the built-in, runtime-integrated tool. Complementary external
tooling lives in the [SeaTunnel
Tools](https://github.com/apache/seatunnel-tools) repository: the [SeaTunnel
Skill](../tools/seatunnel-skill.md) (Claude integration for IDE-level
assistance), the [MCP Server](../tools/seatunnel-mcp.md) (programmatic LLM
access to SeaTunnel resources), and [x2seatunnel](../tools/x2seatunnel.md)
(config conversion).
diff --git a/docs/en/ai-cli/quickstart.md b/docs/en/ai-cli/quickstart.md
new file mode 100644
index 0000000000..27b858106c
--- /dev/null
+++ b/docs/en/ai-cli/quickstart.md
@@ -0,0 +1,114 @@
+---
+sidebar_position: 2
+---
+
+# Quick Start
+
+## Prerequisites
+
+- Python 3.10+ (3.11 or 3.12 recommended)
+- macOS, Linux, or WSL on Windows
+- One LLM provider credential:
+ - **AWS Bedrock** — AWS credentials (profile, env vars, or IAM role)
+ - **Anthropic API** — `ANTHROPIC_API_KEY`
+ - **OpenAI API** (or compatible) — `OPENAI_API_KEY`
+- (Optional) a SeaTunnel installation for engine-level validation and job
execution
+
+## Install
+
+### From the SeaTunnel distribution (recommended)
+
+The shell wrapper installs Python dependencies automatically on first run:
+
+```bash
+# First run — installs dependencies, launches interactive setup
+bin/seatunnel-ai.sh --init
+
+# After init, launch directly
+bin/seatunnel-ai.sh
+```
+
+`SEATUNNEL_HOME` is set automatically to the distribution root.
+
+### From source
+
+```bash
+cd seatunnel-cli
+bash setup.sh # installs all providers + dev tools
+seatunnel --init # interactive provider setup
+```
+
+## Configure a Provider
+
+```bash
+# Option A: AWS Bedrock (default)
+export AI_PROVIDER=bedrock
+export AWS_REGION=us-east-1
+
+# Option A2: OpenAI-family Bedrock models (Responses-API-only, e.g.
gpt-5.6-terra)
+export AI_PROVIDER=bedrock-mantle
+export OPENAI_MODEL='openai.gpt-5.6-terra'
+
+# Option B: Anthropic API
+export AI_PROVIDER=anthropic
+export ANTHROPIC_API_KEY=sk-ant-...
+
+# Option C: OpenAI or compatible
+export AI_PROVIDER=openai
+export OPENAI_API_KEY=sk-...
+# export OPENAI_BASE_URL=https://... # Azure OpenAI, DeepSeek, local vLLM,
...
+```
+
+API keys are read from environment variables only — they are never written to
config files.
+
+## Generate Your First Pipeline
+
+### Single-shot mode
+
+```bash
+seatunnel "Sync MySQL users table to S3 Parquet"
+seatunnel "从 Kafka 读取订单数据写入 ClickHouse" -o my_job.conf
+```
+
+### Interactive mode
+
+```bash
+seatunnel
+```
+
+```
+🐬 SeaTunnel > Sync PostgreSQL orders to Doris
+
+ 📋 Generated SeaTunnel Config
+ Config saved to: .data/last_job.conf
+
+🐬 SeaTunnel > Add a filter to only include orders where amount > 100
+
+ 📋 Generated SeaTunnel Config (updated)
+
+🐬 SeaTunnel > /check
+ [1] Local validation: PASS
+ [2] Engine --check: PASS
+ Dry-run PASSED — Config is ready to execute.
+
+🐬 SeaTunnel > /run
+ Job submitted: 1234567890 (orders-sync)
+ Status: FINISHED
+```
+
+### Useful commands
+
+| Command | Description |
+|---------|-------------|
+| `/check` | Validate the last config; auto-diagnoses and fixes on failure |
+| `/run` | Execute via REST API or `seatunnel.sh`; auto-repairs on failure |
+| `/connectors` | List available sources, sinks, and transforms |
+| `/remember <text>` | Save a non-sensitive fact (hosts, ports, database
names) |
+| `/sessions`, `/resume` | List and resume previous conversations |
+
+## Tips for Good Results
+
+- **Include connection details in the prompt** (host, port, database, table):
the config comes back runnable instead of full of placeholders.
+- **State batch vs real-time intent explicitly** ("one-off full copy" vs
"capture changes continuously") — this drives BATCH/STREAMING and CDC-connector
selection.
+- **Credentials are placeholdered by design**: generated configs reference
`${MYSQL_PASSWORD}`-style variables; export them before `/run`.
+- For scenarios the models are measurably weak at (conditional routing,
PostgreSQL-CDC prerequisites, Doris/StarRocks options — see the
[benchmark](benchmark.md)), review the generated config before running it in
production.
diff --git a/docs/sidebars.js b/docs/sidebars.js
index 217c4c993e..840b430399 100644
--- a/docs/sidebars.js
+++ b/docs/sidebars.js
@@ -348,6 +348,16 @@ const sidebars = {
"edge-agent/faq"
]
},
+ {
+ "type": "category",
+ "label": "AI CLI",
+ "items": [
+ "ai-cli/overview",
+ "ai-cli/quickstart",
+ "ai-cli/design",
+ "ai-cli/benchmark"
+ ]
+ },
{
"type": "category",
"label": "Tools",
diff --git a/docs/zh/ai-cli/benchmark.md b/docs/zh/ai-cli/benchmark.md
new file mode 100644
index 0000000000..96d3b1ec41
--- /dev/null
+++ b/docs/zh/ai-cli/benchmark.md
@@ -0,0 +1,79 @@
+---
+sidebar_position: 4
+---
+
+# 模型基准测试
+
+AI CLI 的准确率靠实测而非假设：专门的基准测试包含 100 个任务、三档复杂度，判定门层层递进直至**真实作业执行**（连接 Docker
化的真实数据源），覆盖 7 个主流大模型。本页汇总方法论、测试结果与选型建议。
+
+> 以下数据于 2026 年 7 月实测，被测对象为 seatunnel-cli v0.1.0（commit `59ada4ec0`），模型由 AWS
Bedrock 提供。模型与 CLI 都在演进，数字是时间快照——需要最新数据请重新运行基准测试。
+
+## 方法论
+
+**任务集**：100 个任务 —— 20 个简单（单源单宿）、45 个中等（类型映射、CDC、transform 链、多表）、35 个复杂（多源
DAG、扇出、条件路由），覆盖 12 个 ETL 场景类别，含 10 个中文任务和 18 个针对已知 LLM
错误模式的规则探针（条件选项误用、BATCH/STREAMING 推断、路由标签接线）。
+
+**判定门**，与 CLI 自身的 check → dry-run → run 管道一一对应：
+
+| 判定门 | 判定方式 | 能抓住什么 |
+|--------|---------|-----------|
+| L1 静态 | HOCON 解析 + 连接器元数据 + 断言 | 语法错误、连接器用错、缺选项 |
+| L3 真实执行 | 在官方 `apache/seatunnel` 镜像里对真实
MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch 运行作业；批任务看退出码，流任务看 60 秒健康存活 |
一切"看着对但跑不通"的问题 |
+
+**修复循环**：判定失败后，把失败层的真实报错喂给 CLI 自带的修复 Agent，最多 3 轮。所有判定均为确定性——没有"LLM 当裁判"。
+
+## 核心发现：静态排名在真实执行下反转
+
+| 模型 | 静态门（L1） | 真实执行（L1+L3） |
+|------|:---:|:---:|
+| Claude Opus 4.8 | 89%（第 3） | **85%（第 1）** |
+| GPT-5.6 Sol | 90%（第 2） | 81%（第 2） |
+| GPT-5.6 Terra | **93%（第 1）** | 74%（第 3） |
+
+静态冠军写出的配置"看着对"但运行时失败最多（静态→真实衰减 -20 个百分点）；静态第三的模型写出的配置真的能跑（仅
-6pp）。**"看着对"和"能跑"是两种不同的模型能力**——这正是基准必须真实执行配置的原因，也是对"只做静态检查就宣称准确率"应保持警惕的原因。
+
+## 静态门完整排名（7 模型）
+
+| 模型 | 通过率（≤5 轮修复） | 首次通过 | 修复救回 |
+|------|:---:|:---:|:---:|
+| GPT-5.6 Terra | 93% | 79% | +14 |
+| GPT-5.6 Sol | 90% | 80% | +10 |
+| Claude Opus 4.8 | 89% | **87%** | +2 |
+| Claude Sonnet 5 | 82% | 78% | +4 |
+| Claude Fable 5 | 80% | 73% | +7 |
+| Qwen3-Coder-Next | 67% | 53% | +14 |
+| DeepSeek V3.2 | 58% | 58% | +0 |
+
+## 模型行为画像
+
+生成能力与修复能力是**两个独立维度**，且家族特征高度一致：
+
+- **一次写对型（Anthropic 系）**：Opus 4.8 首次通过率全场最高（静态 87% / 真实
77%），静态→真实衰减最小——它写出来的基本就能跑。但修复贡献极低（+2）：自己的失败很少能修回来。
+- **迭代修复型（GPT-5.6 Terra）**：首过平平，但修复能力全场最强——真实运行时失败修回 48%，总分从 55% 爬到
74%。它的初稿"看着对但跑不通"的风险最大。
+- **修复失能型（DeepSeek V3.2）**：100
个任务中修复成功次数为零——修复输出要么与原配置相同要么直接放弃。对内建自动修复循环的产品，等于砍掉了一半机制。
+- 两个值得注意的现象：深推理旗舰（Fable 5）只排第 5——在该用合理默认值干活的场景里过度追问澄清；10
个中文任务上，国产模型得分反而更低（Qwen 5/10、DeepSeek 4/10），GPT-5.6 Sol 拿了 10/10。
+
+## 模型选型指南
+
+| 场景 | 推荐 | 理由 |
+|------|------|------|
+| 交互式使用（用户在等） | GPT-5.6 Sol / Terra | 快（25–58 秒/任务），综合准确率高 |
+| 无人值守 / 批量生成 | Claude Opus 4.8 | 首次通过率最高，产出最可能免修改直接运行 |
+| 预算敏感的回归测试 | Qwen3-Coder-Next | 成本最低档；适合冒烟，不适合生产生成 |
+| 自动修复流程中避免使用 | DeepSeek V3.2 | 实测修复成功率为零 |
+
+## 已知弱场景（所有模型）
+
+13 个任务在前三名模型上**全部失败**。如果你的管道属于以下形态，生产使用前请人工复核生成的配置：
+
+| 场景 | 失败根因 |
+|------|---------|
+| Doris / StarRocks 目标端 | 连接器选项知识缺失（fenodes、load 端口、save mode） |
+| PostgreSQL-CDC | 前置条件（复制槽、publication）未在配置中体现 |
+| 条件路由（一个源按谓词拆分到多个目标） | 并行 SQL transform 下的 `plugin_output`/`plugin_input` 接线 |
+| 宽 DAG（5+ 块、混合 transform） | 标签配对与块组织错误 |
+
+这些聚类是工程改进目标，不是永久限制：正在通过连接器元数据注入和黄金样例覆盖逐项解决，改进效果用同一基准验证。
+
+## 修复循环的实测效果
+
+把**真实引擎报错**喂回修复 Agent，可挽回 47% 的运行时失败（前三名模型合计）。同一个模型修复结构化校验错误的成功率约为修复原始 Java
堆栈的 2 倍——证明修复循环下一步最有杠杆的改进是结构化错误解析，而不是更聪明的模型。
diff --git a/docs/zh/ai-cli/design.md b/docs/zh/ai-cli/design.md
new file mode 100644
index 0000000000..ddf7070947
--- /dev/null
+++ b/docs/zh/ai-cli/design.md
@@ -0,0 +1,89 @@
+---
+sidebar_position: 3
+---
+
+# 设计思路
+
+本页说明 AI CLI 如何把一句自然语言变成可运行的配置，以及它为什么被设计成"多智能体 + 分层校验"的流水线，而不是一次 LLM 调用。
+
+## 为什么不是一次 LLM 调用
+
+SeaTunnel 配置生成是一个领域 DSL 问题，有几类"单次生成"无法可靠处理的失败模式：
+
+- **150+ 连接器、每个 20–50 个选项** —— 没有任何模型对每个选项名和类型有完整、最新的知识；
+- **条件选项依赖** —— 比如 text 格式专属的选项出现在 PARQUET 格式的配置里会导致运行时错误；
+- **DAG 接线语义** —— `plugin_output`/`plugin_input` 标签必须在 source/transform/sink
块之间精确配对；
+- **模式推断** —— CDC 源必须是 STREAMING 模式；有界源不应携带流式专属选项。
+
+因此 CLI 用权威连接器元数据为模型"接地"，对每一份候选配置做校验，并用真实报错驱动修复。
+
+## 多智能体流水线
+
+```
+用户输入（自然语言）
+ │
+ ▼
+┌─────────────────┐ ┌──────────────────────┐
+│ Planner Agent │───▶│ 连接器知识库（工具） │
+│ （意图识别 + │◀───│ │
+│ 必要时追问） │ └──────────────────────┘
+└────────┬────────┘
+ │ 结构化计划
+ ▼
+┌─────────────────┐
+│ Config Agent │ 生成 HOCON 配置
+└────────┬────────┘
+ ▼
+┌─────────────────┐ ┌──────────────────────┐
+│ Validator Agent │───▶│ 本地校验 + │
+│ │ │ 引擎 --check │
+└────────┬────────┘ └──────────────────────┘
+ │
+ 通过 ── 是 ─▶ 输出并自动保存
+ │
+ 否（最多 3 轮）
+ ▼
+┌─────────────────┐
+│ Fix Agent │ 修正错误，重新校验
+└─────────────────┘
+
+后续 /check 或 /run 失败时
+ ▼
+┌─────────────────┐
+│ Repair Agent │ 诊断真实引擎/运行时报错，修补配置
+└─────────────────┘
+```
+
+**Planner** 负责意图分类（新建管道 / 提问 / 错误诊断）、选择连接器，只在没有合理默认值时才向用户追问。**Config Agent**
基于注入 prompt 的连接器元数据编写 HOCON。**Validator** 结合确定性本地检查与 LLM 语义审查。**Fix/Repair
Agent** 接收真实错误文本——生成阶段的校验结果，或 `/check`、`/run` 之后的真实引擎堆栈——在原配置上修补，而不是从头重新生成。
+
+## 连接器知识库
+
+两级解析让选项知识保持准确，无需人工维护 prompt 文本：
+
+1. **运行时 API** —— 运行中 SeaTunnel 引擎的 option-rules 接口（始终最新）；
+2. **内置元数据** —— `connector_metadata.json`，通过反射从引擎导出并随 CLI 打包，包含
source/sink/transform 的必填与可选选项、条件选项组和取值约束。
+
+Prompt 按请求动态组装：只注入 Planner 选中的连接器元数据，条件选项按触发条件显式分组（"仅当 `format = text`
时包含"）——这是对抗模型编造选项、误放选项最有效的单项措施。
+
+三层生成策略叠加其上：**Skill 场景手册**（批同步、CDC 实时、多管道等剧本）→ **黄金样例**（常见组合的已验证配置）→
**连接器元数据**（权威选项规则）。
+
+## 校验管道
+
+| 阶段 | 方式 | 能抓住什么 |
+|------|------|-----------|
+| 1. 本地校验 | HOCON 语法、结构、必填项对照元数据、路由标签配对、未解析的 `${VAR}`
占位符（字段级豁免：`file_name_expression` 中的 `${now}` 等引擎模板变量不误报）、安全检查 |
语法错误、缺失/未知选项、接线错误 |
+| 2. 引擎 dry-run | `seatunnel.sh --check` / `--dry-run static` —— 引擎的真实解析路径 |
插件可加载性、选项类型、未知 key、DAG 拓扑 |
+| 3. 真实执行 | `/run` 走 REST API 或 `seatunnel.sh` | 一切运行时问题：连接、schema、CDC 前置条件 |
+
+三层逐级递进；任一层失败都会把该层的真实报错喂给修复循环。
+
+## 设计原则
+
+- **接地，不轻信**：prompt 里每一条连接器知识都来自引擎元数据，不依赖模型记忆。
+- **确定性校验，LLM 修复**：结论由解析器和引擎给出；LLM 只负责生成与修复，从不担任裁判。
+- **真实报错是最好的提示词**：修复 Agent
接收实际校验输出和堆栈。实测证明结构化、具体的错误信息比原始噪声的修复成功率高得多（见[基准测试](benchmark.md)）。
+- **默认安全**：生成的配置对所有凭证使用 `${ENV_VAR}` 占位；会话存储脱敏；`/remember` 拒绝敏感值。
+
+## 用数据说话，不靠假设
+
+这套架构由[专门的基准测试](benchmark.md)持续度量——100 个任务、包含真实执行在内的多层判定门、覆盖 7
个大模型。基准结果直接驱动改进路线（连接器知识注入、修复循环的结构化错误解析），并且未来对 prompt 或元数据的每次修改都可以在同一任务集上做 A/B
对照。
diff --git a/docs/zh/ai-cli/overview.md b/docs/zh/ai-cli/overview.md
new file mode 100644
index 0000000000..e53961264d
--- /dev/null
+++ b/docs/zh/ai-cli/overview.md
@@ -0,0 +1,40 @@
+---
+sidebar_position: 1
+---
+
+# AI CLI 概览
+
+SeaTunnel AI CLI 用自然语言生成可直接投产的 SeaTunnel 数据管道配置。用中文或英文描述一个数据同步任务，CLI 会生成经过校验的
HOCON 配置文件——内置自动校验、错误修复和一键执行。
+
+```
+🐬 SeaTunnel > 把 MySQL 的 users 表同步到 S3，Parquet 格式
+
+ ⚙️ 正在生成 SeaTunnel 配置...
+ ✅ 校验配置（第 1 轮）...
+
+ 📋 已生成 SeaTunnel 配置
+ 配置已保存至: .data/last_job.conf
+```
+
+AI CLI 以 `seatunnel-cli` 模块的形式内置在 SeaTunnel 主仓库中，并随标准发行版一起打包。可通过发行版的
`bin/seatunnel-ai.sh` 启动，也可以从源码用 pip 安装。
+
+## 核心能力
+
+- **自然语言生成配置** —— 中英文输入，输出完整 HOCON 配置
+- **多 LLM 提供商** —— AWS Bedrock（含通过 `bedrock-mantle` 端点接入的 OpenAI
系模型）、Anthropic API、OpenAI 及兼容 API
+- **多智能体流水线** —— Planner → 配置生成 → 校验 → 自动修复，最多 3 轮纠错
+- **连接器知识库** —— 150+ 连接器的完整选项规则与取值约束，来自运行中引擎或内置元数据
+- **校验与修复** —— 本地检查、引擎 `--check`/dry-run，`/check` 或 `/run` 失败时由 LLM 自动诊断修复
+- **会话与记忆** —— 多轮对话细化配置、会话持久化、连接信息记忆（绝不存储凭证）
+
+## 本节文档
+
+| 页面 | 内容 |
+|------|------|
+| [快速开始](quickstart.md) | 安装、配置 LLM 提供商、生成第一条管道 |
+| [设计思路](design.md) | 多智能体架构与校验管道的设计 |
+| [模型基准测试](benchmark.md) | 7 个大模型的实测准确率、选型建议与已知弱场景 |
+
+## 与其它 AI 工具的关系
+
+AI CLI 是内置于运行时的工具。外部配套工具维护在 [SeaTunnel
Tools](https://github.com/apache/seatunnel-tools) 仓库中：[SeaTunnel
Skill](../tools/seatunnel-skill.md)（Claude 集成，IDE 级辅助）、[MCP
服务](../tools/seatunnel-mcp.md)（LLM 编程式访问 SeaTunnel
资源）、[x2seatunnel](../tools/x2seatunnel.md)（配置转换）。
diff --git a/docs/zh/ai-cli/quickstart.md b/docs/zh/ai-cli/quickstart.md
new file mode 100644
index 0000000000..1b73f11906
--- /dev/null
+++ b/docs/zh/ai-cli/quickstart.md
@@ -0,0 +1,114 @@
+---
+sidebar_position: 2
+---
+
+# 快速开始
+
+## 前置条件
+
+- Python 3.10+（推荐 3.11 或 3.12）
+- macOS、Linux 或 Windows WSL
+- 任意一种 LLM 提供商凭证：
+ - **AWS Bedrock** —— AWS 凭证（profile、环境变量或 IAM 角色）
+ - **Anthropic API** —— `ANTHROPIC_API_KEY`
+ - **OpenAI API**（或兼容 API）—— `OPENAI_API_KEY`
+- （可选）SeaTunnel 安装目录，用于引擎级校验和作业执行
+
+## 安装
+
+### 从 SeaTunnel 发行版启动（推荐）
+
+Shell 包装脚本首次运行时会自动安装 Python 依赖：
+
+```bash
+# 首次运行 —— 自动安装依赖并进入交互式初始化
+bin/seatunnel-ai.sh --init
+
+# 初始化完成后直接启动
+bin/seatunnel-ai.sh
+```
+
+`SEATUNNEL_HOME` 会自动指向发行版根目录。
+
+### 从源码安装
+
+```bash
+cd seatunnel-cli
+bash setup.sh # 安装全部提供商依赖 + 开发工具
+seatunnel --init # 交互式配置提供商
+```
+
+## 配置 LLM 提供商
+
+```bash
+# 方式 A：AWS Bedrock（默认）
+export AI_PROVIDER=bedrock
+export AWS_REGION=us-east-1
+
+# 方式 A2：Bedrock 上的 OpenAI 系模型（仅支持 Responses API，如 gpt-5.6-terra）
+export AI_PROVIDER=bedrock-mantle
+export OPENAI_MODEL='openai.gpt-5.6-terra'
+
+# 方式 B：Anthropic API
+export AI_PROVIDER=anthropic
+export ANTHROPIC_API_KEY=sk-ant-...
+
+# 方式 C：OpenAI 或兼容 API
+export AI_PROVIDER=openai
+export OPENAI_API_KEY=sk-...
+# export OPENAI_BASE_URL=https://... # Azure OpenAI、DeepSeek、本地 vLLM 等
+```
+
+API 密钥只从环境变量读取——绝不写入任何配置文件。
+
+## 生成第一条管道
+
+### 单发模式
+
+```bash
+seatunnel "Sync MySQL users table to S3 Parquet"
+seatunnel "从 Kafka 读取订单数据写入 ClickHouse" -o my_job.conf
+```
+
+### 交互模式
+
+```bash
+seatunnel
+```
+
+```
+🐬 SeaTunnel > 把 PostgreSQL 的 orders 表同步到 Doris
+
+ 📋 已生成 SeaTunnel 配置
+ 配置已保存至: .data/last_job.conf
+
+🐬 SeaTunnel > 加个过滤，只保留 amount > 100 的订单
+
+ 📋 已生成 SeaTunnel 配置（已更新）
+
+🐬 SeaTunnel > /check
+ [1] 本地校验: PASS
+ [2] 引擎 --check: PASS
+ Dry-run 通过 —— 配置可以执行。
+
+🐬 SeaTunnel > /run
+ 作业已提交: 1234567890 (orders-sync)
+ 状态: FINISHED
+```
+
+### 常用命令
+
+| 命令 | 说明 |
+|------|------|
+| `/check` | 校验最近生成的配置；失败时自动诊断修复 |
+| `/run` | 通过 REST API 或 `seatunnel.sh` 执行；失败时自动修复 |
+| `/connectors` | 列出可用的 source、sink 和 transform |
+| `/remember <内容>` | 记住非敏感信息（主机、端口、库名等） |
+| `/sessions`、`/resume` | 查看和恢复历史会话 |
+
+## 提问技巧
+
+- **在描述里带上连接信息**（主机、端口、库名、表名）：生成的配置可以直接运行，而不是一堆占位符。
+- **明确批处理还是实时**（"一次性全量拷贝" vs "持续捕获变更"）——这决定 BATCH/STREAMING 模式和是否选用 CDC 连接器。
+- **凭证默认占位**：生成的配置用 `${MYSQL_PASSWORD}` 这类环境变量引用密码，`/run` 前先 export。
+- 对于模型实测偏弱的场景（条件路由拆流、PostgreSQL-CDC 前置配置、Doris/StarRocks
选项——见[基准测试](benchmark.md)），生产使用前建议人工复核生成的配置。
