Claus Ibsen created CAMEL-26960:
-----------------------------------
Summary: camel-jbang - Mark the mistakes in the code of AI
answers, such as unknown options
Key: CAMEL-26960
URL: https://issues.apache.org/jira/browse/CAMEL-26960
Project: Camel
Issue Type: Improvement
Components: camel-jbang
Reporter: Claus Ibsen
Assignee: Claus Ibsen
Fix For: 4.24.0
The AI clients of the Camel CLI show code in their answers: YAML routes,
application.properties lines, endpoint URIs. Nothing checks that code. A file
the model writes goes through the validator, but code in an answer does not,
and a beginner copies it.
*Data* from a benchmark with a local model (Nov 13-14 2026, qwen3.6:35b-mlx, 5
example apps, 53 beginner questions in 3 rounds, so 159 answers per run):
* Without an instruction: 6 unknown options in 4 answers.
* With an instruction in the first message to look options up and to check
every snippet with camel_validate_source: 32 unknown options in 14 answers. The
model wrote 221 code blocks (84 without the instruction) and checked none of
them. Its 3 calls of camel_validate_source checked the route file, not a
snippet.
Examples from the answers:
* {{idempotentConsumer}} with {{messageIdCache}} and {{cacheName}} (the option
is {{idempotentRepository}})
* {{redeliveryPolicy}} with {{multiplier}} (the option is {{backOffMultiplier}})
* {{camel.metrics-enabled}} (the key is {{camel.metrics.enabled}}),
{{camel.tracing}}, {{camel.health.check-routes-on-startup}}
* {{log:ERROR?level=ERROR&message=...}} (the log endpoint has no message option)
* {{onException}} with the exception class as a YAML key
* {{forceOpen}}, {{failureThreshold}} and {{openDuration}} on a circuit breaker
*Proposal: mark the mistakes, do not change the answer.*
* When an answer is complete, take its fenced code blocks (YAML, XML, Java,
properties) and check them with the same checks as camel_validate_source with
content.
* Under each block with a problem, show one line per problem with the nearest
valid names. For example: "multiplier is not an option of redeliveryPolicy; did
you mean backOffMultiplier?"
* Add the findings to the conversation, so the next answer does not repeat the
mistake.
* No extra model call. Always on.
*Where*
* The check itself goes in camel-jbang-core, next to camel_validate_source, so
all clients share it.
* camel ask: print the findings below the answer.
* The AI panel: mark the blocks in the answer, for the built-in model and for
ACP agents (their answers come in the same way).
* External MCP clients never show Camel their answer. They can get a tool that
checks the code blocks of a text. This can be a follow-up.
*Fragments*
Most code in an answer is a fragment, not a whole route. {{camel validate
yaml}} finds {{messageIdCache}} in a whole route. But it refuses a bare
{{redeliveryPolicy:}} block for its shape ("object found, array expected")
before it checks the options. The check needs a fragment mode: check the block
against the schema of its EIP, or wrap a list of steps in a route.
*How to measure*
The benchmark counts unknown options in answers. With this in place, no unknown
option should be shown without a mark (32 in the run with the instruction).
*Later, not in this issue:* a repair turn, where the model gets the findings
and corrects its answer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)