Claus Ibsen created CAMEL-26960:
-----------------------------------

             Summary: camel-jbang - Mark the mistakes in the code of AI 
answers, such as unknown options
                 Key: CAMEL-26960
                 URL: https://issues.apache.org/jira/browse/CAMEL-26960
             Project: Camel
          Issue Type: Improvement
          Components: camel-jbang
            Reporter: Claus Ibsen
            Assignee: Claus Ibsen
             Fix For: 4.24.0


The AI clients of the Camel CLI show code in their answers: YAML routes, 
application.properties lines, endpoint URIs. Nothing checks that code. A file 
the model writes goes through the validator, but code in an answer does not, 
and a beginner copies it.

*Data* from a benchmark with a local model (Nov 13-14 2026, qwen3.6:35b-mlx, 5 
example apps, 53 beginner questions in 3 rounds, so 159 answers per run):
* Without an instruction: 6 unknown options in 4 answers.
* With an instruction in the first message to look options up and to check 
every snippet with camel_validate_source: 32 unknown options in 14 answers. The 
model wrote 221 code blocks (84 without the instruction) and checked none of 
them. Its 3 calls of camel_validate_source checked the route file, not a 
snippet.

Examples from the answers:
* {{idempotentConsumer}} with {{messageIdCache}} and {{cacheName}} (the option 
is {{idempotentRepository}})
* {{redeliveryPolicy}} with {{multiplier}} (the option is {{backOffMultiplier}})
* {{camel.metrics-enabled}} (the key is {{camel.metrics.enabled}}), 
{{camel.tracing}}, {{camel.health.check-routes-on-startup}}
* {{log:ERROR?level=ERROR&message=...}} (the log endpoint has no message option)
* {{onException}} with the exception class as a YAML key
* {{forceOpen}}, {{failureThreshold}} and {{openDuration}} on a circuit breaker

*Proposal: mark the mistakes, do not change the answer.*
* When an answer is complete, take its fenced code blocks (YAML, XML, Java, 
properties) and check them with the same checks as camel_validate_source with 
content.
* Under each block with a problem, show one line per problem with the nearest 
valid names. For example: "multiplier is not an option of redeliveryPolicy; did 
you mean backOffMultiplier?"
* Add the findings to the conversation, so the next answer does not repeat the 
mistake.
* No extra model call. Always on.

*Where*
* The check itself goes in camel-jbang-core, next to camel_validate_source, so 
all clients share it.
* camel ask: print the findings below the answer.
* The AI panel: mark the blocks in the answer, for the built-in model and for 
ACP agents (their answers come in the same way).
* External MCP clients never show Camel their answer. They can get a tool that 
checks the code blocks of a text. This can be a follow-up.

*Fragments*
Most code in an answer is a fragment, not a whole route. {{camel validate 
yaml}} finds {{messageIdCache}} in a whole route. But it refuses a bare 
{{redeliveryPolicy:}} block for its shape ("object found, array expected") 
before it checks the options. The check needs a fragment mode: check the block 
against the schema of its EIP, or wrap a list of steps in a route.

*How to measure*
The benchmark counts unknown options in answers. With this in place, no unknown 
option should be shown without a mark (32 in the run with the instruction).

*Later, not in this issue:* a repair turn, where the model gets the findings 
and corrects its answer.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to