branch: externals/llm
commit 0bea0123af3964502219e7373a25f2409a5e8f01
Author: Andrew Hyatt <[email protected]>
Commit: GitHub <[email protected]>
Add tool choice for forcing tools on or off (#220)
---
NEWS.org | 3 ++-
README.org | 10 +++++++---
llm-claude.el | 12 ++++++++++++
llm-openai.el | 15 +++++++++++++++
llm-test.el | 14 +++++++++++---
llm-vertex.el | 31 ++++++++++++++++++++++---------
llm.el | 24 ++++++++++++++++++++++--
7 files changed, 91 insertions(+), 18 deletions(-)
diff --git a/NEWS.org b/NEWS.org
index 4691fef088..a095905f9d 100644
--- a/NEWS.org
+++ b/NEWS.org
@@ -1,4 +1,5 @@
-* Version 0.27.4
+* Version 0.28.0
+- Add tool calling options, for forbidding or forcing tool choice.
- Fix bug (or perhaps breaking change) in Ollama tool use.
- Add Gemini 3 model, update Gemini code to pass thought signatures
- Add =json-response= capability to Claude 4.5 and 4.1 Opus models
diff --git a/README.org b/README.org
index 0c06ff1cdc..0a2fb4ee12 100644
--- a/README.org
+++ b/README.org
@@ -228,8 +228,6 @@ Conversations can take place by repeatedly calling
~llm-chat~ and its variants.
** Caution about ~llm-chat-prompt-interactions~
The interactions in a prompt may be modified by conversation or by the
conversion of the context and examples to what the LLM understands. Different
providers require different things from the interactions. Some can handle
system prompts, some cannot. Some require alternating user and assistant chat
interactions, others can handle anything. It's important that clients keep to
behaviors that work on all providers. Do not attempt to read or manipulate
~llm-chat-prompt-interactions~ af [...]
** Tool use
-*Note: tool use is currently beta quality. If you want to use tool use,
please watch the =llm=
[[https://github.com/ahyatt/llm/discussions][discussions]] for any
announcements about changes.*
-
Tool use is a way to give the LLM a list of functions it can call, and have it
call the functions for you. The standard interaction has the following steps:
1. The client sends the LLM a prompt with tools it can use.
2. The LLM may return which tools to use, and with what arguments, or text as
normal.
@@ -239,7 +237,7 @@ Tool use is a way to give the LLM a list of functions it
can call, and have it c
This basic structure is useful because it can guarantee a well-structured
output (if the LLM does decide to use the tool). *Not every LLM can handle tool
use, and those that do not will ignore the tools entirely*. The function
=llm-capabilities= will return a list with =tool-use= in it if the LLM supports
tool use. Because not all providers support tool use when streaming,
=streaming-tool-use= indicates the ability to use tool uses in
~llm-chat-streaming~. Right now only Gemini, Vertex, [...]
-The way to call functions is to attach a list of functions to the =tools= slot
in the prompt. This is a list of =llm-tool= structs, which is a tool that is an
elisp function, with a name, a description, and a list of arguments. The
docstrings give an explanation of the format. An example is:
+The way to call tools is to attach a list of tools to the =tools= slot in the
prompt. This is a list of =llm-tool= structs, which is a tool that is an elisp
function, with a name, a description, and a list of arguments. The docstrings
give an explanation of the format. An example is:
#+begin_src emacs-lisp
(llm-chat-async
@@ -276,6 +274,12 @@ Tools will be called with vectors for array results, =nil=
for false boolean res
Be aware that there is no gaurantee that the tool will be called correctly.
While the LLMs mostly get this right, they are trained on Javascript functions,
so imitating Javascript names is recommended. So, "write_email" is a better
name for a function than "write-email".
Examples can be found in =llm-tester=. There is also a function call to
generate function calls from existing elisp functions in
=utilities/elisp-to-tool.el=.
+
+Tool use can be controlled by the =:tool-options= param in
=llm-make-chat-prompt=
+that takes a =llm-tool-options= struct. This can be set to force or forbid
tool
+calling, or to force a specific tool to be called. This is useful when a
+converastion with tools happens and the tools remain constant but how they are
+used may need to change. Ollama does not support currently support this.
** Media input
*Note: media input functionality is currently alpha quality. If you want to
use it, please watch the =llm=
[[https://github.com/ahyatt/llm/discussions][discussions]] for any
announcements about changes.*
diff --git a/llm-claude.el b/llm-claude.el
index 13771b3a8a..ccb7d6d50e 100644
--- a/llm-claude.el
+++ b/llm-claude.el
@@ -109,6 +109,18 @@
(setq request (plist-put request :system system)))
(when (llm-chat-prompt-temperature prompt)
(setq request (plist-put request :temperature
(llm-chat-prompt-temperature prompt))))
+ (when-let* ((options (llm-chat-prompt-tool-options prompt)))
+ (setq request (plist-put request :tool_choice
+ (append
+ (list :type (pcase
(llm-tool-options-tool-choice options)
+ ('auto "auto")
+ ('any "any")
+ ('none "none")
+ ((pred stringp) "tool")
+ (_ (error "Unknown tool choice
option: %s"
+
(llm-tool-options-tool-choice options)))))
+ (when (stringp (llm-tool-options-tool-choice
options))
+ (list :name (llm-tool-options-tool-choice
options)))))))
(when (llm-chat-prompt-reasoning prompt)
(setq request (plist-put request :thinking
(let (thinking-plist)
diff --git a/llm-openai.el b/llm-openai.el
index 80853818af..2b67ae5a10 100644
--- a/llm-openai.el
+++ b/llm-openai.el
@@ -200,6 +200,20 @@ PROVIDER is the Open AI provider struct."
(list :tools (vconcat (mapcar #'llm-provider-utils-openai-tool-spec
(llm-chat-prompt-tools prompt))))))
+(defun llm-openai--build-tool-choice (prompt)
+ "Build the tool_choice field if present in PROMPT."
+ (when-let ((options (llm-chat-prompt-tool-options prompt)))
+ (list :tool_choice
+ (pcase (llm-tool-options-tool-choice options)
+ ('auto "auto")
+ ('none "none")
+ ('any "required")
+ ((pred stringp)
+ (list
+ :function (list :name (llm-tool-options-tool-choice options))
+ :type "function"))
+ (_ (error "Unknown tool choice option: %s"
(llm-tool-options-tool-choice options)))))))
+
(defun llm-openai--build-tool-interaction (interaction)
"Build the tool interaction for INTERACTION."
(mapcar
@@ -301,6 +315,7 @@ STREAMING if non-nil, turn on response streaming."
(llm-openai--build-max-tokens prompt)
(llm-openai--build-response-format prompt)
(llm-openai--build-tools prompt)
+ (llm-openai--build-tool-choice prompt)
(llm-openai--build-messages prompt)))
;; Merge non-standard params
diff --git a/llm-test.el b/llm-test.el
index 9ae1fc0dc5..2e163396c2 100644
--- a/llm-test.el
+++ b/llm-test.el
@@ -210,7 +210,9 @@
:name "func"
:description "desc"
:args '((:name "arg1"
:description "desc1" :type string)
- (:name "arg2"
:description "desc2" :type integer :optional t))))))
+ (:name "arg2"
:description "desc2" :type integer :optional t))))
+ :tool-options (make-llm-tool-options
+ :tool-choice "func")))
:openai
(:model "model"
:messages [(:role "user" :content "Hello world")]
@@ -223,7 +225,8 @@
:properties
(:arg1 (:description "desc1"
:type "string")
:arg2 (:description
"desc2" :type "integer"))
- :required ["arg1"])))])
+ :required ["arg1"])))]
+ :tool_choice (:function (:name "func") :type "function"))
:gemini
(:contents [(:role "user" :parts [(:text "Hello world")])]
:tools [(:function_declarations
@@ -234,7 +237,10 @@
:properties
(:arg1 (:description "desc1"
:type "string")
:arg2 (:description
"desc2" :type "integer"))
- :required ["arg1"]))])])
+ :required ["arg1"]))])]
+ :toolConfig (:functionCallingConfig
+ (:mode "ANY"
+ :allowedFunctionNames ["func"])))
:ollama (:model "model"
:messages [(:role "user" :content "Hello world")]
:tools
@@ -261,6 +267,8 @@
(:arg1 (:description "desc1" :type
"string")
:arg2 (:description "desc2"
:type "integer"))
:required ["arg1"]))]
+ :tool_choice (:type "tool"
+ :name "func")
:stream :false)))
"A list of tests for `llm-provider-chat-request'.")
diff --git a/llm-vertex.el b/llm-vertex.el
index ea92f10a43..1186c3e530 100644
--- a/llm-vertex.el
+++ b/llm-vertex.el
@@ -250,15 +250,28 @@ information than standard tool use."
(when (llm-chat-prompt-tools prompt)
;; Although Gemini claims to be compatible with Open AI's function
declaration,
;; it's only somewhat compatible.
- `(:tools
- [(:function_declarations
- ,(vconcat (mapcar
- (lambda (tool)
- `(:name ,(llm-tool-name tool)
- :description ,(llm-tool-description tool)
- :parameters ,(llm-provider-utils-openai-arguments
- (llm-tool-args tool))))
- (llm-chat-prompt-tools prompt))))]))
+ (append
+ `(:tools
+ [(:function_declarations
+ ,(vconcat (mapcar
+ (lambda (tool)
+ `(:name ,(llm-tool-name tool)
+ :description ,(llm-tool-description tool)
+ :parameters
,(llm-provider-utils-openai-arguments
+ (llm-tool-args tool))))
+ (llm-chat-prompt-tools prompt))))])
+ (when-let* ((options (llm-chat-prompt-tool-options prompt)))
+ `(:toolConfig
+ (:functionCallingConfig
+ ,(append
+ (list :mode (pcase (llm-tool-options-tool-choice options)
+ ('auto "AUTO")
+ ('none "NONE")
+ ('any "ANY")
+ ((pred stringp) "ANY")
+ (_ (error "Unknown tool choice option: %s"
(llm-tool-options-tool-choice options)))))
+ (when (stringp (llm-tool-options-tool-choice options))
+ `(:allowedFunctionNames [,(llm-tool-options-tool-choice
options)]))))))))
(llm-vertex--chat-parameters prompt model)))
;; TODO: remove after September 2025, this is only here so people can upgrade
to
diff --git a/llm.el b/llm.el
index 1b38185ec1..df12629744 100644
--- a/llm.el
+++ b/llm.el
@@ -72,7 +72,18 @@ See %s for the details on the restrictions on use." name
tos)))
Use of this directly is deprecated, instead use `llm-make-chat-prompt'."
context examples interactions tools temperature max-tokens response-format
- reasoning non-standard-params)
+ reasoning non-standard-params tool-options)
+
+(cl-defstruct llm-tool-options
+ "Contains standarized options for tool use.
+
+TOOL-CHOICE is a symbol that indicates how tools are chosen, or a string
+tool name. It can be `none' (no tools are used), `auto' (the LLM
+chooses whether to use a tool), or `any' (the LLM must call one of the
+tools), or the string name of a tool, to force the use of that tool.
+
+Not all providers can force a particular tool use."
+ (tool-choice 'auto))
(cl-defstruct llm-chat-prompt-interaction
"This defines a single interaction given as part of a chat prompt.
@@ -231,7 +242,7 @@ instead."
:args args
:async async))
-(cl-defun llm-make-chat-prompt (content &key context examples tools
+(cl-defun llm-make-chat-prompt (content &key context examples tools
tool-options
temperature max-tokens response-format
reasoning non-standard-params)
"Create a `llm-chat-prompt' with CONTENT sent to the LLM provider.
@@ -272,6 +283,14 @@ optional. When this is given, the LLM will either call the
function or return text as normal, depending on what the LLM
decides.
+TOOL-OPTIONS is a `llm-tool-options' struct, which can be used for
+setting whether tool callins is allowed or not or forcing a tool call to
+be used. This is useful because some models work best when the same
+tools are in all the requests in a conversation, but the program wants
+to have some control over how they are called. LLMs typically have some
+variablility in what tool options can be used, so not all can support
+every option.
+
TEMPERATURE is a floating point number with a minimum of 0, and
maximum of 1, which controls how predictable the result is, with
0 being the most predicatable, and 1 being the most creative.
@@ -331,6 +350,7 @@ vectors (if a list). This is optional."
:content s))
(if (listp content) content (list content)))
:tools tools
+ :tool-options tool-options
:temperature temperature
:max-tokens max-tokens
:response-format response-format
