ChickenBenny opened a new pull request, #1389:
URL: https://github.com/apache/flink-agents/pull/1389

   Linked issue: #144
   
   ### Purpose of change
   
   This PR adds `VectorStoreSearchTool`, which wraps a declared vector store as 
a tool. With it, a ReAct agent's model decides for itself whether to search, 
what to search for, and whether to try again with a different query. Today the 
agent always retrieves context before each chat call, whether the question 
needs it or not. The new docs section and example show how to use it.
   
   #### Runtime flow
   
   1. When you create the tool, a `mode="before"` validator turns `name` and 
`description` into `ToolMetadata`. The model only sees one argument, `query: 
str` (`VectorStoreSearchArgs`). A tool rebuilt from the plan JSON already has 
`metadata`, so it passes straight through. That's what makes the JSON round 
trip work.
   2. The chat model sends this metadata to the model. When the model calls the 
tool, `ToolCallAction` finds it and invokes `call(query=...)`.
   3. `call` looks up `vector_store` through `resource_context` and runs 
`VectorStoreQuery(query_text=query, limit=max_results)`.
   4. It returns `[{"content", "metadata"}, ...]`, or `"No relevant documents 
found."` when nothing matches. `to_tool_response` turns the list into JSON.
   
   #### Key decisions
   
   - **It lives in `api`, not `plan`.** Users declare this tool themselves. 
`BashTool` is different because the framework injects it. It's not exported 
from `api/tools/__init__.py`, same as `Tool` and `FunctionTool`.
   - **Errors are raised, not returned as `ToolResponse.error`.** 
`ToolCallAction` already records the failure and lets `UncapturableException` 
through. The guard for that, `raise_if_uncapturable`, lives in `plan`, so this 
tool can't use it. Also, most errors here are configuration mistakes the model 
can't fix anyway. This differs from what I first proposed in the issue.
   - **The result is JSON, not formatted text.** Metadata in Flink jobs is 
often structured (timestamps, event IDs, nested fields), and returning data for 
the framework to serialize matches how other tools work. The cost is about 
11–17% more tokens.
   - **The tool converts metadata itself.** Otherwise a single `datetime` or 
NaN makes `to_tool_response` fall back to `str()` for the whole result, and the 
model gets a Python repr instead of JSON.
   - **`description` is required.** How well the model uses the tool depends 
almost entirely on it.
   - **No score, id, filters, or collection override for now.** Any of these 
could come in a follow-up.
   
   ### Behavioral Semantics
   
   #### Interaction decisions
   
   What happens with each combination of constructor arguments:
   
   | `metadata` given | `name` / `description` given | Result |
   |---|---|---|
   | no | both | metadata is built from them |
   | no | one or none | `ValidationError` naming what's missing |
   | yes | none | used as is (this is the deserialization path) |
   | yes | either | `ValidationError`, so neither side silently wins |
   
   Which setups work, by vector store and `tool-call.async`:
   
   | Vector store | `tool-call.async` | Flink 1.20.4 and earlier, 2.0.0–2.0.1, 
2.1.0–2.1.2, 2.2.0 | Other Flink versions |
   |---|---|---|---|
   | Python | any | works | works |
   | Java | `false` | works (not tested) | works (not tested) |
   | Java | `true` | not supported; the docs say to set it to `false` | works 
(not tested) |
   
   `context_retrieval_action` falls back to sync for this case on its own. This 
tool can't, because it sits in `api` and can't import `JavaVectorStore` or 
`support_async()`.
   
   #### Behavioral contracts
   
   1. The model sees one required string argument, `query`.
   2. The query is sent as `query_text` with `limit=max_results` (default 3, 
must be positive).
   3. The vector store is looked up by name through the resource context on 
every call.
   4. Each result has exactly `content` and `metadata`, in the order the vector 
store returns them.
   5. When nothing matches, the tool returns `"No relevant documents found."`. 
That's a string, while a hit returns a list. This is on purpose: the model 
shouldn't mistake an empty result for a failed call.
   6. Metadata is converted to JSON-compatible values, so the response is 
always valid JSON.
   7. Errors from looking up or querying the vector store propagate out of 
`call`.
   8. Construction requires `name`, `description` and `vector_store`, rejects 
`metadata` together with `name` or `description`, and doesn't mutate the input 
dict.
   9. Settings survive the agent plan JSON round trip.
   
   #### Failure behavior
   
   | Failure | What happens |
   |---|---|
   | Missing or invalid constructor argument | `ValidationError` when the tool 
is created |
   | `vector_store` isn't declared, or the embedding model or vector store 
fails | raised at call time. The model sees ``Tool `<name>` execute failed.``, 
and the exception text goes to `ToolResponseEvent.error` |
   | `name` doesn't match the resource name | existing framework behavior: the 
model sees ``Tool `<name>` does not exist.`` |
   | A metadata value has no JSON form (custom object, NaN, ±inf) | absorbed: 
converted with `str()` |
   
   ### Tests
   
   | Contract | Tests |
   |---|---|
   | 1 | `test_metadata_exposes_only_query_argument` |
   | 2 | `test_call_queries_with_query_text_and_max_results`, 
`test_max_results_defaults_to_three`, `test_rejects_invalid_arguments` |
   | 3 | `test_call_looks_up_vector_store_by_name` |
   | 4 | `test_call_returns_content_and_metadata_only` |
   | 5 | `test_call_returns_message_when_nothing_matches`, 
`test_tool_response_is_plain_message_when_nothing_matches` |
   | 6 | `test_call_converts_metadata_to_json_compatible_values`, 
`test_call_converts_numpy_metadata_to_native_values`, 
`test_tool_response_is_json_with_non_json_metadata` |
   | 7 | `test_call_propagates_vector_store_errors` |
   | 8 | `test_rejects_invalid_arguments`, 
`test_rejects_metadata_with_name_or_description`, 
`test_does_not_mutate_input_dict` |
   | 9 | `test_tool_survives_agent_plan_json_round_trip`, 
`test_validates_from_dumped_fields` |
   
   The tests in `api/tools/tests/` use a mocked resource context and vector 
store. The ones in `plan/tests/tools/` go through `AgentPlan` JSON and 
`to_tool_response`. The riskiest path is rebuilding the tool from plan JSON, 
and it's covered both by a test and by the example run.
   
   I ran `examples/rag/react_rag_agent_example.py` on a local Flink 2.3 
mini-cluster with Ollama (`qwen3:4b`, `nomic-embed-text`) and Chroma. The model 
called the tool once for "What is Apache Flink Agents?" and didn't call it for 
"What is 2 + 3?".
   
   **Not verified**
   - Java vector stores, with either `tool-call.async` setting.
   - Models other than `qwen3:4b`.
   - How the docs render in Hugo.
   - The full `./tools/ut.sh` run (this PR has no Java or runtime changes).
   
   <details>
   <summary>Implementation invariants and supporting details</summary>
   
   - Pydantic reuses an existing instance of the same class without running 
`mode="before"` validators, so the validator's non-dict branch is just 
defensive (`test_validates_existing_instance`).
   - The validator raises `ValueError` instead of letting `dict.pop` raise 
`KeyError`, so Pydantic wraps it in a `ValidationError`.
   - Metadata conversion is recursive: `dict` keys become strings; `list`, 
`tuple` and `set` become lists; `bytes` become Base64 (same as 
`to_tool_response`); `datetime`, `date` and `time` become ISO strings. Values 
that have `dtype` and `tolist()` (numpy scalars and arrays) go through 
`tolist()`, without importing numpy. That matters because `numpy.int64` and 
`numpy.bool_` aren't subclasses of `int` or `bool`.
   - `tool_type()` returns `FUNCTION`. It's only used as a label in execution 
metadata.
   - Token cost was measured with tiktoken `o200k_base` on chunks of this 
repo's docs, 300 samples per chunk size. Compared with the same fields as plain 
text, JSON costs +17% at 500 characters, +13% at 1000 and +11% at 2000.
   - Tested with Python 3.12 and Pydantic 2.12. `ruff check` and `ruff format 
--check` pass.
   
   </details>
   
   ### API
   
   Adds `flink_agents.api.tools.vector_store_search_tool.VectorStoreSearchTool` 
and `VectorStoreSearchArgs`. Nothing existing changes. It's Python only for 
now: there's no Java counterpart or YAML declaration yet. Both can follow once 
we agree on the Python shape.
   
   ### Documentation
   
   - [ ] `doc-needed`
   - [ ] `doc-not-needed`
   - [x] `doc-included`
   
   Adds a "Use as a Tool" section to the vector stores page.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   - [x] Yes
   - [ ] No
   
   Generated-by: Claude Code 2.1.325 (Claude Opus 5.5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to