[
https://issues.apache.org/jira/browse/CAMEL-25532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125969#comment-18125969
]
Claus Ibsen commented on CAMEL-25532:
-------------------------------------
Benchmark run with the prototype (Oct 10-11 2026; branch
feature/CAMEL-25532-lenient-input, local): the stepwise ladder, qwen3.6 35B, 5
runs of main and 5 runs of the branch on the same main, alternating.
||||main||lenient||
|steps passed (strict)|333/355 (93.8%)|328/355 (92.4%)|
|final state right|338|337|
|runs with every step passed|87/105|83/105|
|tool calls|2,231|2,063 (-7.5%)|
|writes and edits|853|746 (-12.5%)|
|refused writes|231|160 (-31%)|
|missed finds|106|66 (-38%)|
|steps at the 20-call cap|19|20|
* The repairs fired in 40 writes (steps under split 16, placeholder quoted 10,
headerName -> name 10, otherwise -> steps 9, handled as constant 9, expression
as constant 5, ...). The fuzzy window placed 15 edits; each one checked in a
failing step was placed right.
* One failure came from a repair: split: {expression: body} written as constant
"body" - a valid route that splits the text "body" (the validator's hint
suggests the same constant). Every other failure of the lenient runs was the
model's logic or a runtime error.
* For this model at ~94% the pass rate does not move (one step per run less,
inside the run-to-run spread); what goes down is the friction. The steps that
still fail are semantic or runtime problems, and those are what run into the
call cap, not layout loops.
Next:
# A bare Simple name (body, header.x, variable.x...) as a plain expression is
not wrapped as a constant, and the validator's hint for it says simple; column
repairs only for a shift of one or two columns; consider keeping only the
structural repairs (no language choice).
# The same comparison with a weaker local model, where layout loops are a
larger share of the failures.
_Claude Code on behalf of Claus Ibsen_
> camel-jbang-mcp - lenient input and anchored edits, so a model adds or
> changes a few EIPs without getting the YAML layout right
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25532
> URL: https://issues.apache.org/jira/browse/CAMEL-25532
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> In the local-model benchmark (the stepwise examples ladder, qwen3.6 35B
> through the camel-jbang-mcp server), most failed writes are about the layout
> of the YAML, not about Camel. In the r7 run and a later k=1 run: 979 calls of
> camel_write_file / camel_edit_file, 238 refused by the validator, 172 of them
> (72%) only for layout: a key at the wrong level (steps: next to split:,
> parameters: next to to:), unquoted braces ({{placeholder}}, {sku}), a plain
> value where the DSL wants an expression (handled: true), otherwise: as a
> list, an option of redeliveryPolicy written on onException. Of those, about
> 61 are YAML syntax/indentation and about 95 are the shape of the DSL. A model
> that gets one of these wrong often writes the same YAML again and again until
> it runs out of tool calls.
> The stepwise case is the common one: the model adds or changes a few EIPs in
> an existing route, it does not write the whole file. Today it does so with
> find/replace, and the replace text must reproduce the exact indentation of
> the place it goes to.
> Proposal:
> 1. Lenient input: accept the shapes models write (as observed in the
> benchmark traces) and store canonical YAML. Each rule is structural and has
> one reading only (never rename by similarity: the cronService / level quick
> fixes of #27666 show where that goes); an ambiguity is an error with both
> readings; the answer reports what was expanded. Applies to camel_write_file
> and camel_edit_file, and can serve camel validate --fix. JSON is accepted as
> it is valid YAML. The file on disk stays canonical, so the next read shows
> the model the right form.
> 2. Anchored edits in camel_edit_file: the model gives where (a line number
> plus the text of that line, or a selector such as "to kafka:orders" / "log in
> route check-stock") and a fragment of a few EIPs at any indentation; the
> server resolves the anchor to a node (the text is the anchor, the line number
> breaks ties: in 79 route files 67% of the lines that carry a value are unique
> in their file), re-indents the fragment to the node, applies the lenient
> rules in that context, validates and splices it in, leaving the rest of the
> file byte for byte. Modes: at (set options of the node), before / after /
> inside (insert EIPs). A miss or a tie answers with the candidate lines.
> 3. camel_get_files can return numbered lines, so line anchors and the
> validator's "Line N:" messages meet.
> 4. Measure each with a harness arm against the r9 run (Oct 10 2026), with
> qwen3.6 and a weaker local model.
> Analysis and local experiments follow as comments.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
