[
https://issues.apache.org/jira/browse/CAMEL-26960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144746#comment-18144746
]
Claus Ibsen commented on CAMEL-26960:
-------------------------------------
Step 1 done: commit 3d34c81fcef9 on the local branch CAMEL-26960-answer-marks,
not pushed yet. Claus chose marks, no repair turn.
*What is checked.* The code blocks of an answer, in the AI panel and in camel
ask:
* YAML: unknown keys and options, from the YAML DSL schema. A fragment is
wrapped in what it belongs to: a {{from:}} without its route goes in a route
file, a list of steps and a single EIP go in the steps of a route, and
{{redeliveryPolicy:}}, {{resilience4jConfiguration:}} and {{onFallback:}} go
under their EIP.
* Endpoint uris: options the component does not have.
* {{camel.*}} keys of properties lines, in a block of any language (the model
writes them in yaml blocks too).
* A value of the wrong type, only when the validator has a hint for it, such as
{{onException}} written as a list with the exception class as a key.
*What is not marked.* A required option left out, a placeholder, a bare "string
found, object expected", the options of an EIP shown without the EIP
({{slidingWindowSize: 10}}), and keys of other runtimes. Code in an answer is
partial on purpose, and a wrong mark costs trust.
*Where the marks show.*
* The AI panel: a ✗ line under the block, with the line in the block and the
hint of the validator. Copy still copies only the code.
* camel ask: a {{Check:}} line on stderr, so stdout is still only the answer.
* The model is told of the marks, and of a simple expression it left invalid,
with the next question. The answer in the history stays as the model wrote it.
*Measured on the benchmark answers of Nov 13* (q1 + q1b, 318 answers, 305 code
blocks): 53 marks in 27 answers, and each one is a real mistake (read one by
one). The whole run took 0.75 s, validator start included.
*Two validator bugs, found by that run and fixed in the same commit:*
* A commented-out option under {{parameters:}} was reported as the unknown
option '# idempotentRepository'. This also affects camel_validate_source.
* A bean with its {{name:}} and an unknown key ({{parameters:}}) got the hint
"the name is a property, not the key". It now gets "goes under properties: or
constructors:".
*Left for the next steps:*
* Option maps without their key ({{slidingWindowSize: 10}}, 6 blocks), which
needs the definition index of step 2.
* Unknown {{camel.*}} groups ({{camel.tracing.enabled}}, {{camel.profile}}).
The catalog does not know them, but it does not know
{{camel.servlet.mapping.context-path}} of Spring Boot either, so they need care.
* Parse errors: their line and column numbers point into the wrapped block.
* ACP agents, and the markdown mode of camel_validate_source.
_Claude Code on behalf of Claus Ibsen (davsclaus)_
> camel-jbang - Mark the mistakes in the code of AI answers, such as unknown
> options
> ----------------------------------------------------------------------------------
>
> Key: CAMEL-26960
> URL: https://issues.apache.org/jira/browse/CAMEL-26960
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> The AI clients of the Camel CLI show code in their answers: YAML routes,
> application.properties lines, endpoint URIs. Nothing checks that code. A file
> the model writes goes through the validator, but code in an answer does not,
> and a beginner copies it.
> *Data* from a benchmark with a local model (Nov 13-14 2026, qwen3.6:35b-mlx,
> 5 example apps, 53 beginner questions in 3 rounds, so 159 answers per run):
> * Without an instruction: 6 unknown options in 4 answers.
> * With an instruction in the first message to look options up and to check
> every snippet with camel_validate_source: 32 unknown options in 14 answers.
> The model wrote 221 code blocks (84 without the instruction) and checked none
> of them. Its 3 calls of camel_validate_source checked the route file, not a
> snippet.
> Examples from the answers:
> * {{idempotentConsumer}} with {{messageIdCache}} and {{cacheName}} (the
> option is {{idempotentRepository}})
> * {{redeliveryPolicy}} with {{multiplier}} (the option is
> {{backOffMultiplier}})
> * {{camel.metrics-enabled}} (the key is {{camel.metrics.enabled}}),
> {{camel.tracing}}, {{camel.health.check-routes-on-startup}}
> * {{log:ERROR?level=ERROR&message=...}} (the log endpoint has no message
> option)
> * {{onException}} with the exception class as a YAML key
> * {{forceOpen}}, {{failureThreshold}} and {{openDuration}} on a circuit
> breaker
> *Proposal: mark the mistakes, do not change the answer.*
> * When an answer is complete, take its fenced code blocks (YAML, XML, Java,
> properties) and check them with the same checks as camel_validate_source with
> content.
> * Under each block with a problem, show one line per problem with the nearest
> valid names. For example: "multiplier is not an option of redeliveryPolicy;
> did you mean backOffMultiplier?"
> * Add the findings to the conversation, so the next answer does not repeat
> the mistake.
> * No extra model call. Always on.
> *Where*
> * The check itself goes in camel-jbang-core, next to camel_validate_source,
> so all clients share it.
> * camel ask: print the findings below the answer.
> * The AI panel: mark the blocks in the answer, for the built-in model and for
> ACP agents (their answers come in the same way).
> * External MCP clients never show Camel their answer. They can get a tool
> that checks the code blocks of a text. This can be a follow-up.
> *Fragments*
> Most code in an answer is a fragment, not a whole route. {{camel validate
> yaml}} finds {{messageIdCache}} in a whole route. But it refuses a bare
> {{redeliveryPolicy:}} block for its shape ("object found, array expected")
> before it checks the options. The check needs a fragment mode: check the
> block against the schema of its EIP, or wrap a list of steps in a route.
> *How to measure*
> The benchmark counts unknown options in answers. With this in place, no
> unknown option should be shown without a mark (32 in the run with the
> instruction).
> *Later, not in this issue:* a repair turn, where the model gets the findings
> and corrects its answer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)