davsclaus opened a new pull request, #30145:
URL: https://github.com/apache/camel/pull/30145

   A follow-up to #29962 of 
[CAMEL-26932](https://issues.apache.org/jira/browse/CAMEL-26932).
   
   ## Problem
   
   Since #29962 a repeated catalog miss without suggestions gets the note "Ask 
with other arguments instead.". For `camel_catalog_sample` that note replaced 
the hint of CAMEL-26875, which was written for exactly this case: there is no 
sample, so write the file, whose checks say what to change.
   
   In the local-model benchmark of last night (r34), in a Citrus test step, the 
model asked `camel_catalog_sample` for `send request` 10 times. The answer was 
"No sample for 'send request'", and the last 8 answers were the note that said 
to ask with other arguments. The step failed.
   
   ## Change
   
   A tool can give a miss hint (`ToolDescriptor.missHint`). The note of a 
repeated miss without suggestions uses it instead of "Ask with other arguments 
instead.". `camel_catalog_sample` gives one:
   
   ```
   "note": "The answer was: No sample for 'send request'. Write the file now 
with camel_write_file: asking again gives no sample either. The write checks 
the file, and its answer says what to change (for a Citrus test: which actions 
it takes). You asked this 3 times now with the same arguments, and the answer 
is the same each time."
   ```
   
   Not changed:
   - a miss with suggestions ("Use one of the suggestions instead: ...");
   - a miss of a tool without a miss hint, such as `camel_catalog_doc`, whose 
hint is for an answer that was found;
   - the note of a repeated answer that was found.
   
   This gives the miss the hint it was meant to have. It may not be enough on 
its own: in the same night the hint for answers that were found came up 79 
times, and the model mostly asked again.
   
   ## Tests
   
   - `RepeatedToolCallsTest`: a repeated `camel_catalog_sample` miss without 
suggestions gets the miss hint; a miss with suggestions still gets them, and no 
`camel_write_file`; every miss hint is whole sentences.
   - camel-jbang-core, the `commands.ai` tests: 1093 run, 0 failures, 1 skipped.
   
   _Claude Code on behalf of davsclaus_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_016Y4dDi5iQNgtHrDdz2NpwF
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to