SEZ9 opened a new issue, #12999: URL: https://github.com/apache/seatunnel/issues/12999
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Three places in `Orchestrator.process_user_input` let a bad intermediate result continue to the next stage instead of stopping. Each is independently reachable, and together they mean an unstable model output is not contained — it is carried forward and delivered. ### SeaTunnel Version dev (`7604a83ec`) ### SeaTunnel Config n/a — the defects are in the CLI's own orchestration, not in a job config. ### Running Command ```shell seatunnel ai run # or: python -m benchmark.runner --models <file> --preset smoke ``` ### Error Exception See each defect below; two of the three produce no error at all, which is the problem. ### Zeta or Flink or Spark Version n/a ### Java or Scala Version n/a ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) --- ## 1. A clarifying question discards the requirement that preceded it `process_user_input` appends the message to `conversation_history`, and only the planner reads that history. Skill matching, slot filling and the enriched prompt are all built from the latest message: ```python matched_skills = SkillRouter.match(structured_plan, user_input) missing = skill.fill_and_check(user_input, self.memory_store, self.client) enriched_prompt = skill.build_enriched_prompt(user_input, ...) ``` So when the user answers a question, the answer *replaces* the requirement. The benchmark makes this visible: after a question it replies ``` Use sensible defaults for anything unspecified. Do not ask further questions - generate the config now. ``` and that string becomes the entire `## User Request` the generator sees — no source, no sink, no table name. `Orchestrator.pending_question` is declared but never assigned or read, so the continuation it implies was never wired up. ## 2. An empty plan still proceeds to generation `SkillExecutor.fetch_all_metadata` iterates the plan's pipelines, so a plan with none returns `""`: ```python if not sections: return "" ``` Generation then runs with no connector option rules at all, which asks the model to recall option keys from memory — the thing the retrieved rules exist to replace. This is reachable on ordinary input: `_parse_legacy_plan` returns `StructuredPlan(pipelines=[])` whenever it cannot find both a source and a sink. Over a run of 4,863 recorded trajectories, only 35 of 65 sampled generation requests carried a `Connector Option Rules` block. What the user sees is worse than a refusal — the exchange continues, produces nothing usable, and ends with `Failed to generate config.` with no indication of which facts were missing. ## 3. A valid config that answers the wrong request is delivered, and the repair stage cannot correct it A config can be entirely valid and still not do what was asked. Choosing `Jdbc` where the plan resolved the request to `MySQL-CDC` produces HOCON that parses, loads its plugins and runs, so `_run_validator` returns `PASS` and the loop delivers it. Nothing compares the config against the plan. The repair stage could not have corrected it either. Its prompt carries only the config and the validation errors: ```python prompt = f"""The following SeaTunnel config has validation issues. Fix them. ## Current Config: ... ## Validation Errors: {validation_errors} Fix ALL the issues and return the corrected config. Keep all existing correct parts unchanged.""" ``` There is no request and no plan to compare against, and "keep all existing correct parts unchanged" protects a wrong connector precisely because nothing flagged it. ## Why these belong together Individually each is a small gap. Together they form a chain: an under-specified request leads to a question (1) which loses the requirement, which yields an empty plan (2) which generates without option rules, and if a connector comes out wrong (3) neither the validator nor the repair rounds revisit it. The CLI's three gates — static check, dry-run, real execution — all sit *after* this chain, and none of them can see an error of intent. Scope note: this is about the CLI's agent orchestration, not about the SeaTunnel data engine. ## Proposed fixes Three stacked PRs, one per defect, since (3) depends on state introduced by (1): 1. Record the outstanding question and accumulate the reply onto the requirement rather than substituting it. 2. Ask for the source and the destination when the plan has no pipeline, instead of generating from an empty option block. 3. Compare the connectors the plan named against the connectors the config uses, feed a mismatch through the existing repair rounds, and give the repair prompt the request and the plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
