[ 
https://issues.apache.org/jira/browse/CAMEL-26960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144757#comment-18144757
 ] 

Claus Ibsen commented on CAMEL-26960:
-------------------------------------

Update: the unknown {{camel.*}} groups are in PR #29906 too (second commit). A 
key of a group Camel does not have is reported when a Camel key is near it, and 
the message names that key: {{camel.tracing.enabled}} -> 
{{camel.trace.enabled}}, {{camel.profile}} -> {{camel.main.profile}}, 
{{camel.kafka.brokers}} -> {{camel.component.kafka.brokers}}. A key near none 
is left alone, since the user's own placeholder keys can start with camel. as 
well. The check is in the shared properties check, so camel validate properties 
and camel_validate_source get it too. It reports nothing on the 304 camel.* 
lines of the properties files of the camel, camel-examples and 
camel-jbang-examples repositories.

_Claude Code on behalf of Claus Ibsen (davsclaus)_


> camel-jbang - Mark the mistakes in the code of AI answers, such as unknown 
> options
> ----------------------------------------------------------------------------------
>
>                 Key: CAMEL-26960
>                 URL: https://issues.apache.org/jira/browse/CAMEL-26960
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-jbang
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Major
>             Fix For: 4.24.0
>
>
> The AI clients of the Camel CLI show code in their answers: YAML routes, 
> application.properties lines, endpoint URIs. Nothing checks that code. A file 
> the model writes goes through the validator, but code in an answer does not, 
> and a beginner copies it.
> *Data* from a benchmark with a local model (Nov 13-14 2026, qwen3.6:35b-mlx, 
> 5 example apps, 53 beginner questions in 3 rounds, so 159 answers per run):
> * Without an instruction: 6 unknown options in 4 answers.
> * With an instruction in the first message to look options up and to check 
> every snippet with camel_validate_source: 32 unknown options in 14 answers. 
> The model wrote 221 code blocks (84 without the instruction) and checked none 
> of them. Its 3 calls of camel_validate_source checked the route file, not a 
> snippet.
> Examples from the answers:
> * {{idempotentConsumer}} with {{messageIdCache}} and {{cacheName}} (the 
> option is {{idempotentRepository}})
> * {{redeliveryPolicy}} with {{multiplier}} (the option is 
> {{backOffMultiplier}})
> * {{camel.metrics-enabled}} (the key is {{camel.metrics.enabled}}), 
> {{camel.tracing}}, {{camel.health.check-routes-on-startup}}
> * {{log:ERROR?level=ERROR&message=...}} (the log endpoint has no message 
> option)
> * {{onException}} with the exception class as a YAML key
> * {{forceOpen}}, {{failureThreshold}} and {{openDuration}} on a circuit 
> breaker
> *Proposal: mark the mistakes, do not change the answer.*
> * When an answer is complete, take its fenced code blocks (YAML, XML, Java, 
> properties) and check them with the same checks as camel_validate_source with 
> content.
> * Under each block with a problem, show one line per problem with the nearest 
> valid names. For example: "multiplier is not an option of redeliveryPolicy; 
> did you mean backOffMultiplier?"
> * Add the findings to the conversation, so the next answer does not repeat 
> the mistake.
> * No extra model call. Always on.
> *Where*
> * The check itself goes in camel-jbang-core, next to camel_validate_source, 
> so all clients share it.
> * camel ask: print the findings below the answer.
> * The AI panel: mark the blocks in the answer, for the built-in model and for 
> ACP agents (their answers come in the same way).
> * External MCP clients never show Camel their answer. They can get a tool 
> that checks the code blocks of a text. This can be a follow-up.
> *Fragments*
> Most code in an answer is a fragment, not a whole route. {{camel validate 
> yaml}} finds {{messageIdCache}} in a whole route. But it refuses a bare 
> {{redeliveryPolicy:}} block for its shape ("object found, array expected") 
> before it checks the options. The check needs a fragment mode: check the 
> block against the schema of its EIP, or wrap a list of steps in a route.
> *How to measure*
> The benchmark counts unknown options in answers. With this in place, no 
> unknown option should be shown without a mark (32 in the run with the 
> instruction).
> *Later, not in this issue:* a repair turn, where the model gets the findings 
> and corrects its answer.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to