davsclaus commented on code in PR #29904:
URL: https://github.com/apache/camel/pull/29904#discussion_r4577456327
##########
design/cli-short-commands.adoc:
##########
@@ -38,8 +40,11 @@ against 10-40 ms estimated), for a fraction of the build and
release work.
The seconds a daemon saves are small next to the turns of the model. The
reason to build it is that a command that
answers at once can be called after every step. An agent makes fewer mistakes
when it validates after every edit, looks
an option up with `camel doc` instead of guessing it, and checks `camel
status` after every reload. Speed alone
-does not make an agent do that, it has to be told (see the motivation), but
speed makes it cheap to do. With the Java
-client the daemon answers in 50-65 ms, under the 0.1 s at which a response
feels instant, like a native command.
+does not make an agent do that, and telling it to is not enough. In a
benchmark with a local model, an instruction to
+check made the model read more, but it checked none of the code it wrote (see
the motivation). The checks have to run
+whether the model asks for them or not, for example in a hook of the agent
that validates every edited file, and a
+check that runs on every edit is where a fast command pays the most. With the
Java client the daemon answers in
Review Comment:
Split into shorter sentences in 0255ea8f4267.
_Claude Code on behalf of davsclaus_
##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc
and got it right. It s
exist, and validated nothing in 9 answers. A lookup also costs context: the
doc of the file component is 17K
characters, and three docs in one answer forced a compaction of the history.
+Two full runs on the night of 2026-11-13 then asked the same model 53
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was
always "what does this app do". The
+ model read more for it (3.9 tool calls per answer instead of 2.1) and passed
it 12 of 15 times instead of 8. Most
+ of that gain is one app: in the first run, the model answered from the
AI-written summary that the readme tool
+ returned, and left a step out. Since CAMEL-26946 that tool returns the facts
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no
validation per answer. Yet the model wrote
+ 2.6 times as much code in all (221 code blocks instead of 84) and validated
none of it: its 3 calls of
+ `camel_validate_source`, all on the first question, checked the route file
of the app, not a snippet. Options that
+ do not exist went from 6 in 4 answers to 32 in 14, all after the first
question, and "how can I make it better"
Review Comment:
Split into shorter sentences in 0255ea8f4267.
_Claude Code on behalf of davsclaus_
##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc
and got it right. It s
exist, and validated nothing in 9 answers. A lookup also costs context: the
doc of the file component is 17K
characters, and three docs in one answer forced a compaction of the history.
+Two full runs on the night of 2026-11-13 then asked the same model 53
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was
always "what does this app do". The
+ model read more for it (3.9 tool calls per answer instead of 2.1) and passed
it 12 of 15 times instead of 8. Most
+ of that gain is one app: in the first run, the model answered from the
AI-written summary that the readme tool
+ returned, and left a step out. Since CAMEL-26946 that tool returns the facts
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no
validation per answer. Yet the model wrote
+ 2.6 times as much code in all (221 code blocks instead of 84) and validated
none of it: its 3 calls of
+ `camel_validate_source`, all on the first question, checked the route file
of the app, not a snippet. Options that
+ do not exist went from 6 in 4 answers to 32 in 14, all after the first
question, and "how can I make it better"
+ passed 5 of 15 times instead of 12.
Review Comment:
Split into shorter sentences in 0255ea8f4267.
_Claude Code on behalf of davsclaus_
##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc
and got it right. It s
exist, and validated nothing in 9 answers. A lookup also costs context: the
doc of the file component is 17K
characters, and three docs in one answer forced a compaction of the history.
+Two full runs on the night of 2026-11-13 then asked the same model 53
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was
always "what does this app do". The
+ model read more for it (3.9 tool calls per answer instead of 2.1) and passed
it 12 of 15 times instead of 8. Most
+ of that gain is one app: in the first run, the model answered from the
AI-written summary that the readme tool
+ returned, and left a step out. Since CAMEL-26946 that tool returns the facts
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no
validation per answer. Yet the model wrote
+ 2.6 times as much code in all (221 code blocks instead of 84) and validated
none of it: its 3 calls of
+ `camel_validate_source`, all on the first question, checked the route file
of the app, not a snippet. Options that
+ do not exist went from 6 in 4 answers to 32 in 14, all after the first
question, and "how can I make it better"
+ passed 5 of 15 times instead of 12.
+- It cost context and time: 13 compactions of the history instead of 6, and
8.1 s of model time per answer instead
+ of 5.8 s.
+- In all, 128 answers passed instead of 130: 4 more on the first question, 6
fewer on the others.
+
+This is one model and one run each. The changes in code, invented options,
compactions and time are well beyond the
+variance between the rounds; most changes in the pass rate of a single
question are not.
+
+Telling the model to check made it read more and write more, not check what it
wrote. The check of the code in an
+answer has to run whether the model asks for it or not: CAMEL-26960 proposes
to mark the mistakes in the code of an
+answer, such as an option that does not exist, in the AI panel and in `camel
ask`.
Review Comment:
Split into shorter sentences in 0255ea8f4267.
_Claude Code on behalf of davsclaus_
##########
design/cli-short-commands.adoc:
##########
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the
start of the daemon itself.
readiness pushed over the WebSocket transport.
. Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a
response feels instant, so the commands
already feel native. A native client for the main platforms, with the Java
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so
what agents read (the Camel CLI pages,
- the instructions of the MCP server, the system prompt of the AI panel) says
that the short commands are cheap to
- call: look an option up instead of guessing it, asking only for the options
needed (the `optionsFilter` of
- `camel_catalog_doc`) as a whole doc page costs context, and validate every
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an
agent uses the CLI, so what agents read
+ (the Camel CLI pages, the instructions of the MCP server, the system prompt
of the AI panel) says that the short
+ commands are cheap to call: look an option up instead of guessing it, asking
only for the options needed (the
+ `optionsFilter` of `camel_catalog_doc`) as a whole doc page costs context.
Telling is not enough for the checks:
Review Comment:
Split into shorter sentences in 0255ea8f4267.
_Claude Code on behalf of davsclaus_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]