davsclaus opened a new pull request, #29906:
URL: https://github.com/apache/camel/pull/29906

   ## What
   
   CAMEL-26960, step 1. The code in an AI answer is what a user copies, but 
only its simple expressions were checked (CAMEL-24805). A benchmark with a 
local model (qwen3.6:35b, 159 answers per run) told the model to validate every 
snippet: it wrote 2.6 times as many code blocks, validated none of them, and 
options that do not exist went from 6 to 32. So the AI panel and `camel ask` 
now check the code blocks of every answer against the catalog, and mark what 
does not exist.
   
   ## Change
   
   **What is checked** (`AnswerChecks.checkCode`, camel-jbang-core):
   - YAML: unknown keys and options, from the YAML DSL schema.
   - Endpoint uris: options the component does not have.
   - `camel.*` keys of properties lines, in a block of any language.
   - A value of the wrong type, only when the validator has a hint for it, such 
as `onException` written as a list with the exception class as a key.
   
   **Fragments** (`YamlFragment`): most code in an answer is a fragment, which 
the schema refuses for its shape before it looks at the keys. So each one is 
wrapped in what it belongs to:
   - a `from:` without its route goes in a route file;
   - a list of steps, or a single EIP, goes in the steps of a route;
   - `redeliveryPolicy:` goes under an `onException`, and 
`resilience4jConfiguration:` and `onFallback:` under a `circuitBreaker`.
   
   **What is not marked:** code in an answer leaves things out on purpose, and 
a wrong mark costs trust in the others. So none of these are marked:
   - a required option left out, a placeholder, a bare "string found, object 
expected";
   - an unknown key where the wrapping put the fragment, such as options shown 
without their EIP (`slidingWindowSize: 10`);
   - parse errors, whose line and column point into the wrapping.
   
   **Where the marks show:**
   - **AI panel:** a line under the block, `` ✗ line 4: `multiplier` is not an 
option of `redeliveryPolicy` ``, with the hint of the validator when it has 
one. *Copy* still copies only the code.
   - **`camel ask`:** a `Check: code block 1, line 4: ...` line on stderr, so 
stdout is still only the answer.
   - **No correction turn**, as decided on the JIRA. The model is told of the 
marks with the next question, and of a simple expression it did not fix, which 
before was only shown. Its own answer stays in the history as it wrote it.
   
   **Two validator bugs**, found when the check ran over the benchmark answers. 
Both also affected `camel_validate_source`:
   - A commented-out option under `parameters:` was reported as the unknown 
option `'# idempotentRepository'` (`EndpointChecks`).
   - A bean with its `name:` and an unknown key was told "the name is a 
property, not the key" (`SchemaHints`). A `parameters:` key now gets "goes 
under properties: or constructors:".
   
   **Size:** `AiPanel.java` is at 3197 lines, under its `SourceSizeTest` cap of 
3200.
   
   **Docs:** the AI panel page (`camel-jbang-tui-ai.adoc`) and the `camel ask` 
page (`camel-jbang-ai.adoc`).
   
   **Not in this PR**, the next steps on the JIRA:
   - option maps without their key;
   - unknown `camel.*` groups (`camel.tracing.enabled`, `camel.profile`); the 
catalog does not know `camel.servlet.*` of Spring Boot either, so a plain check 
would mark it;
   - the answers of ACP agents;
   - a markdown mode of `camel_validate_source` for MCP clients.
   
   ## Measured
   
   The check ran over the 318 answers of two benchmark runs (305 code blocks). 
It made 53 marks in 27 answers, and I read each one: all are real mistakes. 
Examples:
   - `messageIdRepository` on `idempotentConsumer`;
   - `forceOpen`, `openDuration` and `failureThreshold` on `circuitBreaker`;
   - `multiplier` and `exponentialBackoffMultiplier` on `redeliveryPolicy`;
   - `deleteAfterRead` on `file`, `retryCount` on `direct`;
   - `steps` under `route`;
   - `camel.server.context-path`.
   
   The whole run took 0.75 s, validator start included.
   
   ## Tests
   
   - `AnswerCodeChecksTest` (new, 9 tests), with snippets from the benchmark:
     - marks that must come: from, steps, an EIP, `redeliveryPolicy`, endpoint 
options, properties, `onException` as a list;
     - blocks that must not be marked: a valid route, an option map without its 
key, placeholders, docker-compose, Kubernetes, Spring Boot settings, a message 
body, an unterminated block;
     - where the marks go, and what the model is told.
   - `AiPanelTest`: the mark is under the block, the model is not asked again, 
and the next question carries the note.
   - `AskLoopTest`: the mark is on stderr.
   - `SourceValidatorEndpointTest` and `YamlValidatorPropertyHintTest`: the two 
bug fixes. Both tests fail on the code before the change.
   - Full suites pass: camel-yaml-dsl-validator 370, camel-jbang-core 2999 + 
254, camel-jbang-plugin-tui 2575.
   
   _Claude Code on behalf of Claus Ibsen (davsclaus)_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_012nJsFsjqqFiwroeHRWaKbS
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to