SEZ9 commented on issue #12539: URL: https://github.com/apache/seatunnel/issues/12539#issuecomment-5883598176
Thanks for the detailed report and the reproducible measurements — the run-to-run variance and the fact that `t1_fake_console` passes in isolation are useful data points. One thing I want to confirm before digging in: I could not find `seatunnel_cli/agents.py` or `PLANNER_SYSTEM` in the current checkout. Could you point to the branch or repository where this CLI and the benchmark runner live (and the commit you benchmarked)? Since `--preset` depends on #12534, is the planner code also part of an open PR? To help separate prompt bias from sampling noise, two follow-ups would be helpful: - Does the misrouting reproduce with temperature 0, or with a different model? If it largely disappears at temperature 0, that points more toward sampling than toward the asymmetric framing. - Since `t1_fake_console` passes on its own but fails inside the full preset, could you check whether any state (conversation history, cached context) is carried between tasks in a preset run? Your suggested fix sounds reasonable to me: broaden the PLAN signals to the common ETL verbs, phrase the rule in terms of what the request contains (a source and a destination, or a named config change), and keep the log/stack-trace rule as the hard CHAT override. Comparing `non-config result: chat` counts across `--trials 3` before and after is the right way to validate it given the non-determinism. Happy to review a PR once you have it. <!-- streview-comment:1406 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
