[ 
https://issues.apache.org/jira/browse/CAMEL-26960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144759#comment-18144759
 ] 

Claus Ibsen commented on CAMEL-26960:
-------------------------------------

*Next steps: wait for a twin run before step 2.*

Step 1 (PR #29906) marks what does not exist in route files, fragments with 
their key, endpoint options and {{camel.*}} properties. Over the 318 benchmark 
answers it made 55 marks in 27 answers, all of them real mistakes.

*What step 2 would add, by size and risk:*
* *Word hints*, easy. 19 of the 55 marks have no hint, because the right name 
is too far by edit distance: {{messageIdRepository}} (4 times) for 
{{idempotentRepository}}, {{multiplier}} for {{backOffMultiplier}}, 
{{openDuration}} for {{waitDurationInOpenState}}. The fix is to suggest the 
option of the same EIP that shares a word with the wrong name. This goes in the 
YAML DSL validator, so it helps file writes too.
* *Parse errors*, easy only for blocks checked as they are. 12 of the 209 YAML 
blocks do not parse (indentation, an unquoted colon, a key given twice). For a 
wrapped fragment, the line and column in the validator's message point into the 
wrapping and would have to be rewritten, which is brittle.
* *Option maps without their key*, hard. 31 blocks (15%), such as 
{{slidingWindowSize: 10}} on its own. This needs an index from keys to the 
definitions of the schema. Choosing the definition is where false marks come 
from ({{id}} and {{description}} are in hundreds of them, and endpoint 
{{parameters:}} maps look the same). The keyless blocks looked at so far held 
valid options, so the gain may be small.

*Why wait.* These mistakes are a local model's (qwen3.6:35b). A frontier model 
makes far fewer, though it still invents options that were renamed or removed 
between Camel 2, 3 and 4. Step 1 catches those for any model, at a cost of 
milliseconds. How much of step 2 is only for local models has not been 
measured. The benchmark has a twin run planned: the same questions to the local 
model and to Claude. It is blocked on CAMEL-26962 (the AI panel sends a 
{{temperature}}, which current Claude models reject).

*Plan:*
* Merge step 1.
* Run the twin benchmark, counting the invented options that are left without a 
mark.
* If Claude also invents options in fragments, step 2 pays for every model. If 
only the local model does, step 2 is polish for local models.
* Step 3 (ACP agents, a markdown mode of camel_validate_source) is not affected.

_Claude Code on behalf of Claus Ibsen (davsclaus)_


> camel-jbang - Mark the mistakes in the code of AI answers, such as unknown 
> options
> ----------------------------------------------------------------------------------
>
>                 Key: CAMEL-26960
>                 URL: https://issues.apache.org/jira/browse/CAMEL-26960
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-jbang
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Major
>             Fix For: 4.24.0
>
>
> The AI clients of the Camel CLI show code in their answers: YAML routes, 
> application.properties lines, endpoint URIs. Nothing checks that code. A file 
> the model writes goes through the validator, but code in an answer does not, 
> and a beginner copies it.
> *Data* from a benchmark with a local model (Nov 13-14 2026, qwen3.6:35b-mlx, 
> 5 example apps, 53 beginner questions in 3 rounds, so 159 answers per run):
> * Without an instruction: 6 unknown options in 4 answers.
> * With an instruction in the first message to look options up and to check 
> every snippet with camel_validate_source: 32 unknown options in 14 answers. 
> The model wrote 221 code blocks (84 without the instruction) and checked none 
> of them. Its 3 calls of camel_validate_source checked the route file, not a 
> snippet.
> Examples from the answers:
> * {{idempotentConsumer}} with {{messageIdCache}} and {{cacheName}} (the 
> option is {{idempotentRepository}})
> * {{redeliveryPolicy}} with {{multiplier}} (the option is 
> {{backOffMultiplier}})
> * {{camel.metrics-enabled}} (the key is {{camel.metrics.enabled}}), 
> {{camel.tracing}}, {{camel.health.check-routes-on-startup}}
> * {{log:ERROR?level=ERROR&message=...}} (the log endpoint has no message 
> option)
> * {{onException}} with the exception class as a YAML key
> * {{forceOpen}}, {{failureThreshold}} and {{openDuration}} on a circuit 
> breaker
> *Proposal: mark the mistakes, do not change the answer.*
> * When an answer is complete, take its fenced code blocks (YAML, XML, Java, 
> properties) and check them with the same checks as camel_validate_source with 
> content.
> * Under each block with a problem, show one line per problem with the nearest 
> valid names. For example: "multiplier is not an option of redeliveryPolicy; 
> did you mean backOffMultiplier?"
> * Add the findings to the conversation, so the next answer does not repeat 
> the mistake.
> * No extra model call. Always on.
> *Where*
> * The check itself goes in camel-jbang-core, next to camel_validate_source, 
> so all clients share it.
> * camel ask: print the findings below the answer.
> * The AI panel: mark the blocks in the answer, for the built-in model and for 
> ACP agents (their answers come in the same way).
> * External MCP clients never show Camel their answer. They can get a tool 
> that checks the code blocks of a text. This can be a follow-up.
> *Fragments*
> Most code in an answer is a fragment, not a whole route. {{camel validate 
> yaml}} finds {{messageIdCache}} in a whole route. But it refuses a bare 
> {{redeliveryPolicy:}} block for its shape ("object found, array expected") 
> before it checks the options. The check needs a fragment mode: check the 
> block against the schema of its EIP, or wrap a list of steps in a route.
> *How to measure*
> The benchmark counts unknown options in answers. With this in place, no 
> unknown option should be shown without a mark (32 in the run with the 
> instruction).
> *Later, not in this issue:* a repair turn, where the model gets the findings 
> and corrects its answer.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to