SEZ9 opened a new issue, #12539: URL: https://github.com/apache/seatunnel/issues/12539
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened The planner classifies each request as either `PLAN:` (generate a pipeline config) or `CHAT:` (answer conversationally). It sometimes classifies an explicit pipeline-creation request as conversation, so the CLI answers with prose and never produces a config. In the benchmark this surfaces as `generation_error: non-config result: chat`. Measured on the 12-task `smoke` preset, `claude-haiku-4.5` via Bedrock, `--level l1`: | Run | Misrouted to CHAT | |-----|-------------------| | 1 | `t1_fake_console`, `t1_mysql_localfile_csv` (2/12) | | 2 | `t1_fake_console`, `t1_mysql_localfile_csv`, `t2_mt_table_list` (3/12) | | single-task run of `t1_fake_console` | none — it passed | At 2-3 of 12 this is a larger failure cluster than any config-quality problem in that preset, and it is non-deterministic: the set differs between runs, and `t1_fake_console` passes when run on its own. The three prompts involved are unambiguous creation requests: - `Create a simple test job: generate 10 rows of fake data with fields id (bigint) and name (string), and print them to the console. Batch mode.` - `Export the users table from MySQL (host localhost, port 3306, database shop, user root) to local CSV files under /tmp/st_bench/users_csv. Batch mode.` - `Use a single Jdbc source with a table_list to read both shop.users and shop.products from MySQL (host localhost, port 3306, user root) and print all rows to the console. Batch mode.` ### Hypothesis `PLANNER_SYSTEM` in `seatunnel_cli/agents.py` frames the decision asymmetrically: ``` Output **PLAN:** ONLY when the user explicitly asks to CREATE or MODIFY a data pipeline config. Signals: "sync X to Y", "read from X write to Y", "create a job that...", "add a transform to...", "modify the config to...", "change the sink to..." Output **CHAT:** for EVERYTHING else, including: - Greetings, help requests, "what is X" questions - **Error logs, stack traces, exception messages** ... (nine more bullets) IMPORTANT: When the user pastes logs or error messages, ALWAYS treat it as a diagnostic request (CHAT), never as a pipeline creation request (PLAN). ``` `ONLY` plus a short literal signal list, against `EVERYTHING else` plus a long emphatic list, biases the classifier toward CHAT. The signal list covers `sync`, `read from`/`write to`, `create a job`, `add a transform`, `modify the config` and `change the sink`, and misses ordinary ETL verbs that mean exactly the same thing — `export`, `import`, `load`, `dump`, `copy`, `migrate`, `ingest`, `archive`, `replicate`. Two of the three failing prompts lead with `Export` and `Use ... to read`, neither of which appears as a signal, so they fall into "EVERYTHING else". This is a hypothesis, not a confirmed root cause — part of the variance is sampling, since the same prompt routes differently across runs. Note this is distinct from the known over-clarification loss (the planner asking `ask_user` instead of generating), which shows up separately as `clarification_asked`. ### What you expected to happen A request that names a source and a sink and asks for data to be moved produces a config, deterministically. ### How to reproduce ```shell python -m benchmark.runner --preset smoke --level l1 \ --provider bedrock --model us.anthropic.claude-haiku-4-5-20251001-v1:0 \ --out /tmp/smoke # then inspect the generation_error column grep chat /tmp/smoke/summary.csv ``` `--preset` requires #12534. ### Suggested fix Broaden the PLAN signal set to cover the common ETL verbs, and state the decision rule by what the request *contains* (a source and a destination, or a named config change) rather than by matching example phrasings. Keep the log/stack-trace rule as the one hard CHAT override, since that one is correct and load-bearing for diagnosis. Because the failure is non-deterministic, any fix needs multiple trials per arm to verify: `--preset smoke --trials 3` before and after, comparing the `non-config result: chat` count. ### SeaTunnel Version dev ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
