gnodet-bot commented on code in PR #29904:
URL: https://github.com/apache/camel/pull/29904#discussion_r4577471705


##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc 
and got it right. It s
 exist, and validated nothing in 9 answers. A lookup also costs context: the 
doc of the file component is 17K
 characters, and three docs in one answer forced a compaction of the history.
 
+Two full runs on the night of 2026-11-13 then asked the same model 53 
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look 
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of 
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was 
always "what does this app do". The
+  model read more for it (3.9 tool calls per answer instead of 2.1) and passed 
it 12 of 15 times instead of 8. Most
+  of that gain is one app: in the first run, the model answered from the 
AI-written summary that the readme tool
+  returned, and left a step out. Since CAMEL-26946 that tool returns the facts 
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no 
validation per answer. Yet the model wrote
+  2.6 times as much code in all (221 code blocks instead of 84) and validated 
none of it. Its 3 calls of
+  `camel_validate_source`, all on the first question, checked the route file 
of the app, not a snippet. Options that
+  do not exist went from 6 in 4 answers to 32 in 14, all after the first 
question. "How can I make it better" passed
+  5 of 15 times instead of 12.
+- It cost context and time: 13 compactions of the history instead of 6, and 
8.1 s of model time per answer instead
+  of 5.8 s.
+- In all, 128 answers passed instead of 130: 4 more on the first question, 6 
fewer on the others.
+
+This is one model and one run each. The changes in code, invented options, 
compactions and time are well beyond the
+variance between the rounds; most changes in the pass rate of a single 
question are not.

Review Comment:
   ⚠️ **Sentence too long (~31 words):** Two independent observations joined by 
a semicolon. Split into two sentences:
   
   ```suggestion
   This is one model and one run each. The changes in code, invented options, 
compactions and time are well beyond the
   variance between the rounds. Most changes in the pass rate of a single 
question are not.
   ```



##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc 
and got it right. It s
 exist, and validated nothing in 9 answers. A lookup also costs context: the 
doc of the file component is 17K
 characters, and three docs in one answer forced a compaction of the history.
 
+Two full runs on the night of 2026-11-13 then asked the same model 53 
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look 
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of 
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was 
always "what does this app do". The
+  model read more for it (3.9 tool calls per answer instead of 2.1) and passed 
it 12 of 15 times instead of 8. Most
+  of that gain is one app: in the first run, the model answered from the 
AI-written summary that the readme tool
+  returned, and left a step out. Since CAMEL-26946 that tool returns the facts 
of the routes by default.
+- On the questions after it, the instruction faded: 0.10 lookups and no 
validation per answer. Yet the model wrote
+  2.6 times as much code in all (221 code blocks instead of 84) and validated 
none of it. Its 3 calls of
+  `camel_validate_source`, all on the first question, checked the route file 
of the app, not a snippet. Options that
+  do not exist went from 6 in 4 answers to 32 in 14, all after the first 
question. "How can I make it better" passed
+  5 of 15 times instead of 12.
+- It cost context and time: 13 compactions of the history instead of 6, and 
8.1 s of model time per answer instead
+  of 5.8 s.
+- In all, 128 answers passed instead of 130: 4 more on the first question, 6 
fewer on the others.
+
+This is one model and one run each. The changes in code, invented options, 
compactions and time are well beyond the
+variance between the rounds; most changes in the pass rate of a single 
question are not.
+
+Telling the model to check made it read more and write more, not check what it 
wrote. The check of the code in an
+answer has to run whether the model asks for it or not. CAMEL-26960 proposes 
to mark the mistakes in the code of an
+answer, such as an option that does not exist, in the AI panel and in `camel 
ask`.

Review Comment:
   ⚠️ **Sentence too long (~29 words):** Split after the first clause to give 
each idea its own sentence:
   
   ```suggestion
   answer has to run whether the model asks for it or not. CAMEL-26960 proposes 
to mark the mistakes in an answer
   (such as options that do not exist) in the AI panel and in `camel ask`.
   ```



##########
design/cli-short-commands.adoc:
##########
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the 
start of the daemon itself.
    readiness pushed over the WebSocket transport.
 . Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a 
response feels instant, so the commands
   already feel native. A native client for the main platforms, with the Java 
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so 
what agents read (the Camel CLI pages,
-  the instructions of the MCP server, the system prompt of the AI panel) says 
that the short commands are cheap to
-  call: look an option up instead of guessing it, asking only for the options 
needed (the `optionsFilter` of
-  `camel_catalog_doc`) as a whole doc page costs context, and validate every 
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an 
agent uses the CLI. So what agents read
+  (the Camel CLI pages, the instructions of the MCP server, the system prompt 
of the AI panel) says that the short

Review Comment:
   ⚠️ **Sentence too long (~31 words):** The "So what agents read … says" 
construction spans two lines. Split the "So" clause into its own sentence:
   
   ```suggestion
   . Tell the agents, and check for them. Speed alone does not change how an 
agent uses the CLI. So the instructions
     agents read (the Camel CLI pages, the instructions of the MCP server, the 
system prompt of the AI panel) say that
     commands are cheap to call. Look an option up instead of guessing it, and 
ask only for the options
   ```



##########
design/cli-short-commands.adoc:
##########
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the 
start of the daemon itself.
    readiness pushed over the WebSocket transport.
 . Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a 
response feels instant, so the commands
   already feel native. A native client for the main platforms, with the Java 
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so 
what agents read (the Camel CLI pages,
-  the instructions of the MCP server, the system prompt of the AI panel) says 
that the short commands are cheap to
-  call: look an option up instead of guessing it, asking only for the options 
needed (the `optionsFilter` of
-  `camel_catalog_doc`) as a whole doc page costs context, and validate every 
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an 
agent uses the CLI. So what agents read
+  (the Camel CLI pages, the instructions of the MCP server, the system prompt 
of the AI panel) says that the short
+  commands are cheap to call. Look an option up instead of guessing it, and 
ask only for the options needed (the
+  `optionsFilter` of `camel_catalog_doc`), since a whole doc page costs 
context. Telling is not enough for the checks:
+  told to validate every snippet, the local model wrote more code and 
validated none of it (see the motivation). So
+  the checks run without the agent's choice. The AI panel and `camel ask` mark 
the mistakes in the code of an answer
+  (CAMEL-26960). The MCP server already validates a file before 
`camel_write_file` writes it. An agent that edits
+  files with its own tools needs a hook that runs `camel validate yaml` on 
each edited route file, and the Camel CLI

Review Comment:
   ⚠️ **Sentence too long (~30 words):** Split at the conjunction to separate 
the hook description from the reference to the pages:
   
   ```suggestion
     files with its own tools needs a hook that runs `camel validate yaml` on 
each edited route file.
     The Camel CLI pages show one. That hook runs on every edit, so it is where 
the daemon saves the most.
   ```



##########
design/cli-short-commands.adoc:
##########
@@ -326,15 +356,23 @@ daemon or a native front does not serve too, and the 
start of the daemon itself.
    readiness pushed over the WebSocket transport.
 . Keep the Java client. At 50-65 ms it answers under the 0.1 s at which a 
response feels instant, so the commands
   already feel native. A native client for the main platforms, with the Java 
client as the fallback, is optional.
-. Tell the agents. Speed alone does not change how an agent uses the CLI, so 
what agents read (the Camel CLI pages,
-  the instructions of the MCP server, the system prompt of the AI panel) says 
that the short commands are cheap to
-  call: look an option up instead of guessing it, asking only for the options 
needed (the `optionsFilter` of
-  `camel_catalog_doc`) as a whole doc page costs context, and validate every 
snippet before answering.
+. Tell the agents, and check for them. Speed alone does not change how an 
agent uses the CLI. So what agents read
+  (the Camel CLI pages, the instructions of the MCP server, the system prompt 
of the AI panel) says that the short
+  commands are cheap to call. Look an option up instead of guessing it, and 
ask only for the options needed (the
+  `optionsFilter` of `camel_catalog_doc`), since a whole doc page costs 
context. Telling is not enough for the checks:
+  told to validate every snippet, the local model wrote more code and 
validated none of it (see the motivation). So
+  the checks run without the agent's choice. The AI panel and `camel ask` mark 
the mistakes in the code of an answer
+  (CAMEL-26960). The MCP server already validates a file before 
`camel_write_file` writes it. An agent that edits
+  files with its own tools needs a hook that runs `camel validate yaml` on 
each edited route file, and the Camel CLI
+  pages show one. That hook runs on every edit, so it is where the daemon 
saves the most.
 . Measure the agents, not only the commands. The local-model benchmark runs in 
the AI panel of the TUI, whose tools
-  run in the process, so it measures the instruction, not the daemon: run it 
with and without the instruction, and
-  compare the pass rate, the calls and the invented options per answer, the 
compactions and the time of the model.
-  The daemon needs a benchmark that drives `camel` through a shell. Make the 
daemon the default only after a release
-  as opt-in, and when that benchmark shows the gain.
+  run in the process, so it measures the instruction and the checks, not the 
daemon. It compares the pass rate, the
+  calls and the invented options per answer, the compactions and the time of 
the model. The first run had the
+  instruction in the first message (see the motivation). The next run puts 
such a line in the system prompt of the
+  AI panel, where it may last past the first question (two jars that differ 
only in that line). A run after that uses

Review Comment:
   ⚠️ **Sentence too long (~32 words):** The parenthetical makes this hard to 
parse. Split after the first clause:
   
   ```suggestion
     instruction in the first message (see the motivation). The next run puts 
such a line in the system prompt of the
     AI panel, where it may last past the first question. The two jars differ 
only in that line. A run after that uses
   ```



##########
design/cli-short-commands.adoc:
##########
@@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc 
and got it right. It s
 exist, and validated nothing in 9 answers. A lookup also costs context: the 
doc of the file component is 17K
 characters, and three docs in one answer forced a compaction of the history.
 
+Two full runs on the night of 2026-11-13 then asked the same model 53 
questions about 5 example apps, 3 times each.
+The second run added an instruction at the start of each conversation: look 
options up with `camel_catalog_doc`, and
+check every snippet with `camel_validate_source` before you answer. Per run of 
159 answers, without and with the
+instruction:
+
+- The first question of each conversation, the one with the instruction, was 
always "what does this app do". The
+  model read more for it (3.9 tool calls per answer instead of 2.1) and passed 
it 12 of 15 times instead of 8. Most
+  of that gain is one app: in the first run, the model answered from the 
AI-written summary that the readme tool
+  returned, and left a step out. Since CAMEL-26946 that tool returns the facts 
of the routes by default.

Review Comment:
   ⚠️ **Sentence too long (~30 words):** The colon joins two independent ideas. 
Split at the colon:
   
   ```suggestion
     of that gain is one app. In the first run, the model answered from the 
AI-written summary that the readme tool
     returned, and left a step out. Since CAMEL-26946 that tool returns the 
facts of the routes by default.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to