kaxil opened a new pull request, #74736: URL: https://github.com/apache/airflow/pull/74736
A `HookToolset` with a wide `allowed_methods` sends every method's name, description and argument schema on every model request, whether the run uses them or not. Pydantic AI already solves this: every toolset has `.defer_loading()`, which hides its tools until the model searches for them, and the `ToolSearch` capability that does the searching is added to every agent automatically (both since 2.36.0, this provider's floor). This PR documents that for the Common AI toolsets, with an example Dag, rather than adding anything to them. New page: **Toolsets > Progressive tool disclosure** (`docs/toolsets/progressive_disclosure.rst`), with `example_progressive_tool_disclosure.py` pulled in through `exampleinclude`. It covers hiding a whole toolset, keeping the common tools visible by passing the names to hide, the native-search and local `search_tools` paths, and when it does not help. Measured on the first model request's tool definitions (tiktoken `o200k_base`, a model without native tool search): | Toolset | All tools sent | `.defer_loading()` | `s3_list_keys` + `s3_read_key` visible | |---|---|---|---| | The example's nine `S3Hook` methods | 1,059 | 163 | 694 | | 25 `S3Hook` methods | 4,609 | 163 | 761 | | `SQLToolset` (four tools) | 488 | 163 | | Both example Dags ran under `airflow dags test` on `claude-haiku-5-5` against a moto S3 bucket and succeeded. The fully deferred one made `search_tools -> s3_list_keys -> s3_read_key -> s3_read_key` in 4 requests; the one with two tools visible made the same S3 calls without a search in 3. The page quotes both task logs. ## Design rationale **Why docs and not a `defer_loading=` argument on the toolsets.** It would duplicate a method every Pydantic AI toolset already has, and add a public argument to keep before 1.0 for no new behaviour; the per-tool choice already exists through `.defer_loading(tool_names)`. What needed checking was that our wrappers do not break it, and they do not: a revealed tool's result is still masked (`AirflowToolset.call_tool` and `MaskingCapability` sit below the wrapper), its calls are still logged by `ToolLoggingCapability`, a templated connection ID still renders through the wrapper, and with `durable=True` a retry after a search replays the steps that followed it and runs only the failed one. Only `search_tools` itself goes unlogged, which `operators/agent.rst` and `toolsets/logging.rst` already said and now say for revealed tools too. **`SQLToolset` and `AgentSkillsToolset` are listed as cases where it does not help.** `SQLToolset` has four tools whatever the database holds (488 tokens vs 163 for `search_tools`), so deferring saves little and adds a search request. `AgentSkillsToolset`'s cost is the skill catalog in the instructions (2,129 tokens for 30 skills against 907 for its tools), which `.defer_loading()` does not touch. **The `skills` extra keeps `pydantic-ai-skills`.** The `pyproject.toml` comment was waiting on pydantic/pydantic-ai#5230, which has shipped. `pydantic-ai-harness`'s `Skills` capability now loads Agent Skills as on-demand capabilities, but only their `SKILL.md` instructions, with no skill resources and no `run_skill_script`, so it does not replace the toolset. The comment, the skills page and the toolset's row in `stability.rst` (still experimental) now say that instead. ## Gotchas - Which search path a model gets is decided from a list of model-name prefixes in the installed Pydantic AI. 2.48 lists `claude-haiku-4-5` but no Haiku 5 model, so `claude-haiku-5-5` takes the local `search_tools` path; the page says so. - A name passed to `.defer_loading()` that matches no tool is ignored without an error, so a typo leaves that tool visible. Also on the page. ## Known issues `HookToolset` decodes base64 for `bytes` parameters but does not convert arguments for `datetime` parameters. In an earlier run of the example, the model called `s3_list_keys` with `from_datetime` as an ISO string and `S3Hook` raised `TypeError: '<=' not supported between instances of 'datetime.datetime' and 'str'`, failing the task. This is independent of `.defer_loading()` and also applies to the existing `HookToolset` example, which exposes `list_keys`; the fix belongs in `HookToolset`'s argument handling, in a separate PR. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
