[
https://issues.apache.org/jira/browse/CAMEL-25541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Federico Mariani reassigned CAMEL-25541:
----------------------------------------
Assignee: Federico Mariani
> camel-jbang-mcp - camel_catalog_doc component answer: bounded option lists,
> an explicit answer when nothing matches, and the page sections that decide
> correctness
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25541
> URL: https://issues.apache.org/jira/browse/CAMEL-25541
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang-mcp
> Affects Versions: 4.22.1
> Reporter: Federico Mariani
> Assignee: Federico Mariani
> Priority: Major
>
> Three findings from an evaluation of coding agents (Claude Code and IBM Bob)
> on Mule-to-Camel migration tasks with camel-jbang-mcp 4.22.1 concern the same
> answer, the component branch of {{camel_catalog_doc}}
> ({{dsl/camel-jbang/camel-jbang-core/.../commands/ai/CatalogDocs.java}}), and
> are fixed together here. CAMEL-25540 and CAMEL-25542 were merged into this
> issue.
> h3. 1. The option list has no bound (15-35 K characters by default)
> {{putOptions}} ({{CatalogDocs.java:1753-1781}}) adds every option the
> {{OptionScope}} accepts; {{common}} only drops deprecated and advanced
> options, so activemq still has 88 options (about 38 K of JSON), file 75 (35
> K), smtp 76 (24 K), http 60 (19 K). Recorded sizes with 4.22.1:
> {noformat}
> camel_catalog_component_doc activemq 32 959 chars (includeOptions=all: 84
> 212, rejected by the client)
> camel_catalog_component_doc file 31 423
> camel_catalog_component_doc smtp 20 611
> camel_component_properties activemq 123 893 "exceeds maximum allowed
> tokens"
> camel_component_properties platform-http 14 996, called 35 times to find
> the HTTP port, which is camel.server.* (CAMEL-25533)
> {noformat}
> 121 of 192 sessions got at least one result over 20 000 characters; Claude
> Code rejected 4 and wrote 48 to a file instead of showing them to the model.
> h3. 2. Nothing matches, and the answer says nothing
> 25 of 81 calls with {{optionsFilter}} came back with {{"matchedOptions": 0}}
> and empty lists, and the agents read that as "unsupported": on smtp with
> {{optionsFilter: tls}} (15 sessions) several decided "no TLS support"; on
> xslt with {{optionsFilter: param}} the agents concluded XSLT takes no
> parameters and one moved to FreeMarker; on activemq {{redeliver}} found
> nothing. On main the filter also searches descriptions, groups and labels,
> which does not close it: smtp + {{tls}} returns only
> {{useJavaMailSessionPropertiesFromHeaders}}, the option whose own description
> warns against this use, while the supported way
> ({{mail.smtp.starttls.enable=true}} as a URI parameter, which the catalog
> knows as the multi-value prefix option {{additionalJavaMailProperties}}) is
> not shown; xslt + {{param}} still matches nothing. (A third case,
> {{schemaValidation}} with {{includeOptions: common}} finding nothing because
> the option is {{advanced}}, is fixed on main: the filter now searches all
> options.)
> Names that still do not resolve on main: the page names
> {{camel_catalog_docs}} returns ({{wireTap-eip}}, {{jackson-dataformat}}) end
> in "Artifact not found" because the
> {{-eip}}/{{-component}}/{{-dataformat}}/{{-language}} suffix is not stripped
> ({{catalogDoc}}, {{:187-300}}); {{retry}} resolves to
> {{optimisticLockRetryPolicy}} by a title word ({{eipByTerm}}, {{:431-440}},
> {{suggestEipNames(term, 1)}}) instead of pointing at the error handler;
> {{camel_catalog_example_file}} with an unknown example answers "Example not
> found: X" without the names ({{ExampleTools.java:113}}).
> h3. 3. The page sections that decide correctness are never in the answer
> {{docExcerpt}} ({{CatalogDocs.java:138}}, {{DOC_EXCERPT_BUDGET = 1400}})
> carries the first {{==}} section only. Three behaviours that decide whether a
> route is correct live further down the page and were never seen by any
> session:
> * xslt: "By default, all headers and variables are added as parameters"
> (section "Getting Usable Parameters into the XSLT", line 129 of 442). One
> session rewrote the transformation in FreeMarker.
> * mail: the Subject/From/To headers are ignored unless
> {{useHeaderSubject}}/{{useHeaderFrom}}/{{useHeaderRecipients}} is true
> (section "Opting out of header overrides", line ~300). 11 sessions sent a
> mail with an empty subject. The option description itself is CAMEL-25534.
> * platform-http under {{camel run}}: {{camel.server.port}} needs
> {{camel.server.enabled=true}}. Only {{platform-http-main.adoc}} says so and
> {{platform-http-component.adoc}} has no {{xref}} to it, so {{docPages}}
> ({{componentDocPages}}, {{:1432}}) is empty. 5 of 10 sessions on one task
> listened on 8080 after 15-47 calls each.
> h3. Fix
> # *Bound the default answer*: without {{optionsFilter}}, return the required
> options plus the first N common options (N about 25), with {{truncated:
> true}}, the total count and a hint ("optionsFilter=<word> finds one;
> includeOptions=all with a limit pages the rest"). Alternatively names only
> (name, type, default) when no filter is given, descriptions for filtered
> calls. Optionally a hard cap on any catalog tool result (16 000 characters)
> with a truncation marker.
> # *Say when nothing matches*: {{"note": "no option matches 'tls'"}} plus the
> closest option names by edit distance (the catalog has the suggester since
> CAMEL-24666) with their group; the multi-value prefix options of the
> component ({{additionalJavaMailProperties}}: "any mail.* URI parameter is
> passed to JavaMail"), also when the filter matched something.
> # *Make the page discoverable*: return the page's section titles
> ({{"sections": [...]}}, with the ones whose text contains the filter term
> first) and a way to fetch one ({{docPage=<section>}}), so a model that sees
> an empty filter can still find the section that answers it. Optionally a
> short curated {{notes}} list per component for defaults that silently change
> behaviour (mail useHeader*, xslt header parameters), returned regardless of
> {{optionsFilter}}.
> # *Resolve the names the tools themselves hand out*: strip a {{-component}},
> {{-dataformat}}, {{-language}} or {{-eip}} suffix and look up again
> (case-insensitive); do not resolve a bare word such as {{retry}} to an EIP by
> a title word alone, return it as a suggestion with the error handler entries
> (or add {{retry}}/{{redelivery}} as aliases of the error handler); list the
> closest example names on "Example not found".
> # *platform-http-component.adoc*: add a Camel Main / {{camel run}} entry to
> "Platform HTTP Provider" (the first section, so it lands in the excerpt) with
> {{camel.server.enabled=true}} + {{camel.server.port}}, and
> {{xref:others:platform-http-main.adoc[...]}}, which {{componentDocPages}}
> already turns into {{docPage=main}}.
> # *Drop {{camel_component_properties}}* rather than add {{filter}}/{{limit}}
> to it: its description says endpoint options "can be set as component-level
> defaults via the same prefix", which is false
> ({{AbstractCamelCatalog.validateConfigurationProperty}} checks
> {{camel.component.<scheme>.*}} against the component options only,
> {{:1221}}), so the tool emits keys that {{camel_configuration_validate}} and
> {{camel run}} reject. A {{configKey}} on each component option in
> {{camel_catalog_doc}} covers the valid half.
> Related: CAMEL-25539 (EIP/language answer: placement and YAML shapes),
> CAMEL-25534, CAMEL-25533, CAMEL-25040 (the excerpt), CAMEL-25283
> ({{docPages}}), CAMEL-23475 (the scopes). Merged here: CAMEL-25540,
> CAMEL-25542.
> h3. Added 2026-10-14: full-key lookup and token numbers from the evaluation
> * {{camel_catalog_doc name=server}} reaches {{mainOptionsGroup}}
> ({{CatalogDocs.java:606}}) and answers the {{camel.server.*}} keys, but the
> forms an agent actually asks for do not: a full key ({{camel.server.port}}
> contains a dot after the group, so {{mainOptionsGroup}} returns null), a
> component plus a word ({{platform-http}} + {{optionsFilter: port}}), and
> {{camel.jbang.*}} ({{catalog.jbangModel()}} is not consulted by any tool).
> Accept a full key and answer the group with the matching option first, and
> route {{camel.jbang.*}} to the jbang model. In the evaluation the port
> question alone cost 30 knowledge searches, 12 {{camel_component_properties
> platform-http}} calls (15 K each, none naming it) and {{Run.class}}
> disassembly in 35 plain-arm trials.
> * Token numbers (Claude Code, 45 Haiku 5.5 trials in the MCP arm, 414 M
> tokens): about 9% of all tokens are re-reads of MCP results on later model
> calls, 4% of them from the 45 results over 20 000 characters (39 of those
> from {{camel_catalog_component_doc}}, 4 from {{camel_component_properties}});
> 125 of 564 results, 21% of all result bytes, were spilled to a file by the
> client and read back with grep. Repeated identical calls: 9%. Sonnet made 0-2
> MCP calls per trial, so for it the bound changes nothing; CAMEL-25646 does.
> _Claude Code on behalf of Croway_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
