[
https://issues.apache.org/jira/browse/CAMEL-25676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127420#comment-18127420
]
Claus Ibsen commented on CAMEL-25676:
-------------------------------------
Notes from the measurement:
* *Both clients missed the retries of ORD-1002.* The errors tool lists only the
exchange that failed in the end (ORD-1003). The two redeliveries of ORD-1002,
which later succeeded, show only as WARN lines in the log and in the failed
count of payment-provider (3 of 5). Neither client read the log, so neither
could explain those failures, or why checkout was slow (two redeliveries, one
second apart). The guidelines could say: to tell what happened to messages,
read the log; when a route's failed count is higher than the errors listed, the
rest were retried or handled. The errors tool could say the same in its answer.
* *camel ask quoted a log line that does not exist* ("Payment for ORD-1002
declined") without having read the log, and on that basis said ORD-1002 was
parked.
* *The example's header comment was wrong:* it said the error handler retries
the first order and parks both, and camel ask quoted it when it read the file.
Fixed in https://github.com/apache/camel/pull/27885.
* *Markdown in the terminal:* the prompt asks for plain text without markdown,
but the model answered with headings, bold and tables, which camel ask prints
as they are. Render the markdown for the terminal, or stop asking for plain
text.
* *One-shot use pays a JVM start per question:* about 96 s of wall time for the
four questions, against the 89 s camel ask reports. Interactive mode keeps one
process.
* *Same model and context window for both:* the llama3.2 default of camel ask
is a placeholder, and LlmClient picks the first installed qwen3.6 (here
qwen3.6:35b-mlx). Both ask Ollama for the same num_ctx, so the model was not
reloaded between them.
> camel ask: confirm risky tool calls, and fit the tools to the model
> -------------------------------------------------------------------
>
> Key: CAMEL-25676
> URL: https://issues.apache.org/jira/browse/CAMEL-25676
> Project: Camel
> Issue Type: Improvement
> Components: camel-jbang
> Reporter: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> {{camel ask}} runs the shared AI tools of camel-jbang-core ({{ToolRegistry}},
> the tools the Camel MCP server serves) against a running integration, with a
> local (Ollama) or a hosted model. It has two gaps: it asks before nothing,
> and it sends every tool to every model.
> h3. 1. No confirmations
> The developer tools section of the security model (security-model.adoc) sets
> the guardrails for Camel's AI assistants: each file write goes through a
> confirmation with the diff, the developer confirms a call that reaches beyond
> the integration, may destroy data or runs code, and an HTTP request goes only
> to the integration's own server. {{camel ask}} has none of the confirmations:
> {{ToolContext}} has no confirmation hook, and {{AskTools}} runs each call as
> it comes.
> * {{camel_write_file}}, {{camel_edit_file}} and its own {{write_file}} write
> files without asking. {{write_file}} writes anywhere under the current
> directory, without the YAML and properties validation of {{camel_write_file}}.
> * {{send_message}} sends to any endpoint, {{execute_sql}} runs any statement,
> {{eval_expression}} and {{camel_eval_expression}} run expressions in
> languages that run code, such as Groovy, and {{stop_application}} stops the
> integration.
> * {{cli_exec}} runs any camel command; its description only asks the model to
> confirm with the user first.
> * {{http_request}} already reaches only the integration's own server.
> Proposal, the same for local and hosted models:
> * Ask before the same calls, by the same rules: a message to an endpoint the
> integration does not have (its routes, its own endpoints and the components
> the catalog marks internal, such as direct:, seda: and log:, stay inside), a
> message body read from a file outside the project folder, SQL other than a
> single SELECT, SHOW or DESCRIBE, a row update, and an expression in a
> language that runs code. Move these rules ({{ToolConfirms}}) to
> camel-jbang-core next to the shared tools, so every client that runs the
> tools applies the same ones.
> * Show the diff of each write and ask. Drop {{write_file}}:
> {{camel_write_file}} and {{camel_edit_file}} cover it, with validation.
> * Ask before {{stop_application}}, the {{camel_control}} actions that stop or
> restart routes or the integration, and a {{cli_exec}} command that changes
> something (such as run, stop, cmd and infra run). Read-only commands (get,
> catalog, version, help) run without asking.
> * Without a terminal to ask on (one-shot use in a script, stdin not a TTY),
> refuse such a call with a message the model can pass on, unless an option
> allows it explicitly, for example {{--yes}}.
> * Add {{camel ask}} to the developer tools section of the security model.
> h3. 2. Every tool to every model
> {{AskTools}} sends all 68 shared tools plus its own 6 ({{cli_list_commands}},
> {{cli_command_help}}, {{cli_exec}}, {{list_files}}, {{read_file}},
> {{write_file}}) with every request: 74 tool definitions. Several do the same
> job, for example {{get_errors}} and {{camel_get_errors}}, {{eval_expression}}
> and {{camel_eval_expression}}, {{catalog_component_doc}} and
> {{camel_catalog_doc}}, {{read_file}} and {{camel_get_files}}. The system
> prompt is generic: nothing on the YAML DSL shapes or Simple, and it points
> the model to a stop_route tool that does not exist.
> Proposal:
> * Local providers ({{LlmClient.isLocalEndpoint}}) get the core tools
> ({{ToolDescriptor.isCore}}) and the tool groups of the selected integration
> ({{get_tool_groups}} works them out). Hosted providers get the full set,
> without the duplicates. An option chooses (auto, core, full), auto by default.
> * Use the authoring guidelines the shared tools are written for (YAML step
> shapes, Simple operators, the edit loop of camel_get_files, camel_edit_file
> and camel_write_file), shared with the other clients instead of a prompt of
> its own, and drop the stop_route line.
> * Allow 25 tool-calling rounds per question instead of 10, and check the
> Simple expressions in the answer ({{AnswerChecks}}), as the edit tools check
> the ones they write.
> h3. Measured
> Oct 15 2026: qwen3.6:35b-mlx on Ollama 0.40.3 on a Mac mini, the
> fail-well/error-handling example running, four questions about it, each in a
> fresh conversation and asked once. Compared with the same model given the
> core tool set (25 tools) and the authoring guidelines:
> || ||core set (25 tools)||camel ask (74 tools)||
> |first request of a question|5.2k tokens|9.5k tokens|
> |time for the four questions|68 s|89 s|
> |model requests|17|11|
> |tokens for the four questions|122k|145k|
> |tool calls|16|12|
> Both got the declined order, the one recorded error and the retry settings
> right. Both named the slowest route but not the reason (two redeliveries of
> one order, one second apart). camel ask also said that one order failed and
> was parked, while it was charged on its third attempt. A hosted model was not
> measured: the full tool set suits it better, and the confirmations apply to
> both.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
