[
https://issues.apache.org/jira/browse/CAMEL-25532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125874#comment-18125874
]
Claus Ibsen commented on CAMEL-25532:
-------------------------------------
Analysis and local experiments (Oct 10 2026, qwen3.6 35B, all benchmark runs so
far).
*Traces* (345 model runs, 6,879 steps, 108,949 tool calls)
* camel_edit_file: 11,159 calls, 19% missed (the find text is not in the file),
23% refused by the validator.
* A miss is cheap today: the next write or edit works 72% of the time (the file
window of CAMEL-24909); only 99 of 1,828 steps with a miss had two or more in a
row. In 69% of the misses a distinctive line of the find is in the file and
unique: the model remembers the line, not the lines around it.
* Refusals split by the fix rate of the next attempt. Named shapes are fixed
next time 95-100% (handled, otherwise, library case, timerName, headerName).
Layout and quoting loop: an option at the column of its EIP 38%, an expression
as a plain string 23%, steps/parameters at the wrong level 16-41%, a list item
in the wrong column 54%, quoting 35-39%. These loops are what end a step at the
tool-call cap.
* Of the refusals in the recent runs (r5-r7), 44-45% have only errors that a
structural rule with one reading would fix. Most of the rest are wrong names
(log level, setHeader headerName, setProperty propertyName), which a curated
alias table from the traces can cover; renaming by similarity cannot (cron=...
renamed to cronService=..., see #27666).
*Experiments* (no tools, JSON answer, temperature 0.7; 42 ladder steps where
the route changes, from the reference pass; validated with the camel-jbang-mcp
validator in a copy of the example directory)
Arms: A = find/replace with today's matching (exact, then by trimmed lines,
re-indented). B = anchored edit (line and text, mode
after/before/inside/replace, a fragment from column 0, re-indented, top-level
entries put at the top level). F = A's prompt, plus a fuzzy window when both
matches miss (the window of the file that best matches the find lines, if it
holds a distinctive line of the find and is clearly best).
One shot, the file just read (126 edits per arm):
||arm||placed||no layout errors||valid||valid + the reference lines||valid with
lenient rules (estimate)||
|A|95%|80%|38%|25%|54%|
|B|87%|69%|29%|20%|40%|
|F|98%|81%|38%|25%|56%|
Chain: the steps of an example one after another in one conversation, the file
not shown again, an invalid result refused (70 edits per arm):
||arm||1st edit placed / valid||later edits placed / valid||
|A|93% / 37%|50% / 18%|
|B|83% / 17%|82% / 15%|
|F|100% / 37%|70% / 25%|
*Reading*
* The familiar format wins. Find/replace is what models know; the anchored
format costs validity and placement on a fresh file (an unfamiliar format:
anchors such as a markdown fence, overlapping edits, replace over several
nodes). Its strength, edits from memory, is mostly recovered by the fuzzy
window, and the misses left refer to a file that never existed (the model's own
refused change), which only showing the file fixes, as the file window does.
* Lenient input is the biggest lever: valid from 38% to an estimated 54-56% in
one shot. Rules by need: an option at the column of its EIP (12), a top-level
entry at the wrong place or order (8), a list where a map is expected or the
other way round (10: otherwise, choice, onFallback, post), a list item or key
in the wrong column (6), YAML parse errors (6), enum case (4). Aliases by need:
unmarshal jackson -> json with library Jackson (10), setProperty
propertyName/property -> name (12), noErrorHandler: true -> noErrorHandler: {}
(6), onException exceptionType -> exception (4).
* Caveats: one model; no tools, catalog or feedback within a step; small
samples (later chain edits: 40 per arm, differences under about 15 points are
noise); the lenient effect is estimated from the error classes, the rules are
not implemented yet.
*Revised scope*
# Lenient input: structural rules with one reading plus a curated alias table,
both from the traces; each expansion reported; the file stays canonical.
# camel_edit_file: a fuzzy window when the exact and the trimmed match miss,
reporting the lines it matched.
# Anchored edits: not as the default edit format; keep line + text anchors for
a later experiment (set an option on a named node), measured against 1 and 2.
_Claude Code on behalf of Claus Ibsen_
> camel-jbang-mcp - lenient input and anchored edits, so a model adds or
> changes a few EIPs without getting the YAML layout right
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25532
> URL: https://issues.apache.org/jira/browse/CAMEL-25532
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> In the local-model benchmark (the stepwise examples ladder, qwen3.6 35B
> through the camel-jbang-mcp server), most failed writes are about the layout
> of the YAML, not about Camel. In the r7 run and a later k=1 run: 979 calls of
> camel_write_file / camel_edit_file, 238 refused by the validator, 172 of them
> (72%) only for layout: a key at the wrong level (steps: next to split:,
> parameters: next to to:), unquoted braces ({{placeholder}}, {sku}), a plain
> value where the DSL wants an expression (handled: true), otherwise: as a
> list, an option of redeliveryPolicy written on onException. Of those, about
> 61 are YAML syntax/indentation and about 95 are the shape of the DSL. A model
> that gets one of these wrong often writes the same YAML again and again until
> it runs out of tool calls.
> The stepwise case is the common one: the model adds or changes a few EIPs in
> an existing route, it does not write the whole file. Today it does so with
> find/replace, and the replace text must reproduce the exact indentation of
> the place it goes to.
> Proposal:
> 1. Lenient input: accept the shapes models write (as observed in the
> benchmark traces) and store canonical YAML. Each rule is structural and has
> one reading only (never rename by similarity: the cronService / level quick
> fixes of #27666 show where that goes); an ambiguity is an error with both
> readings; the answer reports what was expanded. Applies to camel_write_file
> and camel_edit_file, and can serve camel validate --fix. JSON is accepted as
> it is valid YAML. The file on disk stays canonical, so the next read shows
> the model the right form.
> 2. Anchored edits in camel_edit_file: the model gives where (a line number
> plus the text of that line, or a selector such as "to kafka:orders" / "log in
> route check-stock") and a fragment of a few EIPs at any indentation; the
> server resolves the anchor to a node (the text is the anchor, the line number
> breaks ties: in 79 route files 67% of the lines that carry a value are unique
> in their file), re-indents the fragment to the node, applies the lenient
> rules in that context, validates and splices it in, leaving the rest of the
> file byte for byte. Modes: at (set options of the node), before / after /
> inside (insert EIPs). A miss or a tie answers with the candidate lines.
> 3. camel_get_files can return numbered lines, so line anchors and the
> validator's "Line N:" messages meet.
> 4. Measure each with a harness arm against the r9 run (Oct 10 2026), with
> qwen3.6 and a weaker local model.
> Analysis and local experiments follow as comments.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
