ChickenBenny opened a new pull request, #1389:
URL: https://github.com/apache/flink-agents/pull/1389
Linked issue: #144
### Purpose of change
This PR adds `VectorStoreSearchTool`, which wraps a declared vector store as
a tool. With it, a ReAct agent's model decides for itself whether to search,
what to search for, and whether to try again with a different query. Today the
agent always retrieves context before each chat call, whether the question
needs it or not. The new docs section and example show how to use it.
#### Runtime flow
1. When you create the tool, a `mode="before"` validator turns `name` and
`description` into `ToolMetadata`. The model only sees one argument, `query:
str` (`VectorStoreSearchArgs`). A tool rebuilt from the plan JSON already has
`metadata`, so it passes straight through. That's what makes the JSON round
trip work.
2. The chat model sends this metadata to the model. When the model calls the
tool, `ToolCallAction` finds it and invokes `call(query=...)`.
3. `call` looks up `vector_store` through `resource_context` and runs
`VectorStoreQuery(query_text=query, limit=max_results)`.
4. It returns `[{"content", "metadata"}, ...]`, or `"No relevant documents
found."` when nothing matches. `to_tool_response` turns the list into JSON.
#### Key decisions
- **It lives in `api`, not `plan`.** Users declare this tool themselves.
`BashTool` is different because the framework injects it. It's not exported
from `api/tools/__init__.py`, same as `Tool` and `FunctionTool`.
- **Errors are raised, not returned as `ToolResponse.error`.**
`ToolCallAction` already records the failure and lets `UncapturableException`
through. The guard for that, `raise_if_uncapturable`, lives in `plan`, so this
tool can't use it. Also, most errors here are configuration mistakes the model
can't fix anyway. This differs from what I first proposed in the issue.
- **The result is JSON, not formatted text.** Metadata in Flink jobs is
often structured (timestamps, event IDs, nested fields), and returning data for
the framework to serialize matches how other tools work. The cost is about
11–17% more tokens.
- **The tool converts metadata itself.** Otherwise a single `datetime` or
NaN makes `to_tool_response` fall back to `str()` for the whole result, and the
model gets a Python repr instead of JSON.
- **`description` is required.** How well the model uses the tool depends
almost entirely on it.
- **No score, id, filters, or collection override for now.** Any of these
could come in a follow-up.
### Behavioral Semantics
#### Interaction decisions
What happens with each combination of constructor arguments:
| `metadata` given | `name` / `description` given | Result |
|---|---|---|
| no | both | metadata is built from them |
| no | one or none | `ValidationError` naming what's missing |
| yes | none | used as is (this is the deserialization path) |
| yes | either | `ValidationError`, so neither side silently wins |
Which setups work, by vector store and `tool-call.async`:
| Vector store | `tool-call.async` | Flink 1.20.4 and earlier, 2.0.0–2.0.1,
2.1.0–2.1.2, 2.2.0 | Other Flink versions |
|---|---|---|---|
| Python | any | works | works |
| Java | `false` | works (not tested) | works (not tested) |
| Java | `true` | not supported; the docs say to set it to `false` | works
(not tested) |
`context_retrieval_action` falls back to sync for this case on its own. This
tool can't, because it sits in `api` and can't import `JavaVectorStore` or
`support_async()`.
#### Behavioral contracts
1. The model sees one required string argument, `query`.
2. The query is sent as `query_text` with `limit=max_results` (default 3,
must be positive).
3. The vector store is looked up by name through the resource context on
every call.
4. Each result has exactly `content` and `metadata`, in the order the vector
store returns them.
5. When nothing matches, the tool returns `"No relevant documents found."`.
That's a string, while a hit returns a list. This is on purpose: the model
shouldn't mistake an empty result for a failed call.
6. Metadata is converted to JSON-compatible values, so the response is
always valid JSON.
7. Errors from looking up or querying the vector store propagate out of
`call`.
8. Construction requires `name`, `description` and `vector_store`, rejects
`metadata` together with `name` or `description`, and doesn't mutate the input
dict.
9. Settings survive the agent plan JSON round trip.
#### Failure behavior
| Failure | What happens |
|---|---|
| Missing or invalid constructor argument | `ValidationError` when the tool
is created |
| `vector_store` isn't declared, or the embedding model or vector store
fails | raised at call time. The model sees ``Tool `<name>` execute failed.``,
and the exception text goes to `ToolResponseEvent.error` |
| `name` doesn't match the resource name | existing framework behavior: the
model sees ``Tool `<name>` does not exist.`` |
| A metadata value has no JSON form (custom object, NaN, ±inf) | absorbed:
converted with `str()` |
### Tests
| Contract | Tests |
|---|---|
| 1 | `test_metadata_exposes_only_query_argument` |
| 2 | `test_call_queries_with_query_text_and_max_results`,
`test_max_results_defaults_to_three`, `test_rejects_invalid_arguments` |
| 3 | `test_call_looks_up_vector_store_by_name` |
| 4 | `test_call_returns_content_and_metadata_only` |
| 5 | `test_call_returns_message_when_nothing_matches`,
`test_tool_response_is_plain_message_when_nothing_matches` |
| 6 | `test_call_converts_metadata_to_json_compatible_values`,
`test_call_converts_numpy_metadata_to_native_values`,
`test_tool_response_is_json_with_non_json_metadata` |
| 7 | `test_call_propagates_vector_store_errors` |
| 8 | `test_rejects_invalid_arguments`,
`test_rejects_metadata_with_name_or_description`,
`test_does_not_mutate_input_dict` |
| 9 | `test_tool_survives_agent_plan_json_round_trip`,
`test_validates_from_dumped_fields` |
The tests in `api/tools/tests/` use a mocked resource context and vector
store. The ones in `plan/tests/tools/` go through `AgentPlan` JSON and
`to_tool_response`. The riskiest path is rebuilding the tool from plan JSON,
and it's covered both by a test and by the example run.
I ran `examples/rag/react_rag_agent_example.py` on a local Flink 2.3
mini-cluster with Ollama (`qwen3:4b`, `nomic-embed-text`) and Chroma. The model
called the tool once for "What is Apache Flink Agents?" and didn't call it for
"What is 2 + 3?".
**Not verified**
- Java vector stores, with either `tool-call.async` setting.
- Models other than `qwen3:4b`.
- How the docs render in Hugo.
- The full `./tools/ut.sh` run (this PR has no Java or runtime changes).
<details>
<summary>Implementation invariants and supporting details</summary>
- Pydantic reuses an existing instance of the same class without running
`mode="before"` validators, so the validator's non-dict branch is just
defensive (`test_validates_existing_instance`).
- The validator raises `ValueError` instead of letting `dict.pop` raise
`KeyError`, so Pydantic wraps it in a `ValidationError`.
- Metadata conversion is recursive: `dict` keys become strings; `list`,
`tuple` and `set` become lists; `bytes` become Base64 (same as
`to_tool_response`); `datetime`, `date` and `time` become ISO strings. Values
that have `dtype` and `tolist()` (numpy scalars and arrays) go through
`tolist()`, without importing numpy. That matters because `numpy.int64` and
`numpy.bool_` aren't subclasses of `int` or `bool`.
- `tool_type()` returns `FUNCTION`. It's only used as a label in execution
metadata.
- Token cost was measured with tiktoken `o200k_base` on chunks of this
repo's docs, 300 samples per chunk size. Compared with the same fields as plain
text, JSON costs +17% at 500 characters, +13% at 1000 and +11% at 2000.
- Tested with Python 3.12 and Pydantic 2.12. `ruff check` and `ruff format
--check` pass.
</details>
### API
Adds `flink_agents.api.tools.vector_store_search_tool.VectorStoreSearchTool`
and `VectorStoreSearchArgs`. Nothing existing changes. It's Python only for
now: there's no Java counterpart or YAML declaration yet. Both can follow once
we agree on the Python shape.
### Documentation
- [ ] `doc-needed`
- [ ] `doc-not-needed`
- [x] `doc-included`
Adds a "Use as a Tool" section to the vector stores page.
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: Claude Code 2.1.325 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]