ShawnGarello opened a new pull request, #1331: URL: https://github.com/apache/flink-agents/pull/1331
Linked issue: #1216 ### Purpose of change The Anthropic integration now sends images and documents returned by a tool to the model, in Java and Python. A tool result holding media becomes one `tool_result` block whose `content` keeps the result's text, image and document blocks in order. Before, any media in a tool result failed the request. Tools can return screenshots, charts or documents the model must read before it continues, and Anthropic accepts them inside `tool_result.content`. This is the first part of the plan in the issue; the other integrations still reject media in tool results. The fields below cover the adapter change; the docs part adds a tool-result media table to the providers page and updates the Tool Use warning. #### Runtime flow Java: `AnthropicChatUtils.convertToAnthropicMessage` passes a tool message to `convertToolResult`. A result of only `TextBlock`s is sent as `content(String)`, the concatenated text, as before. Otherwise each block becomes a `ToolResultBlockParam.Content.Block` in order: `ofText`, `ofImage` through `toImageBlockParam` (now shared with user messages), or `ofDocument` through the existing `toDocumentBlockParam`. Other media throws. Python: `convert_to_anthropic_message` already built the `tool_result` content from the result's blocks; its media check now accepts tool messages too, so `_to_media_block` runs for them. #### Key decisions - Tool results use the user-message helpers, so one place decides which media Anthropic accepts. - Text-only results keep their wire form, so existing requests don't change. - No dependency change: anthropic-java 2.12.0 has `ToolResultBlockParam.Content.Block.ofImage/ofDocument`, and the Python adapter sends plain dicts. ### Behavioral Semantics #### Interaction decisions | Message | Text only | Supported media | Unsupported media | Undecodable Base64 plain text | |---|---|---|---|---| | User | unchanged | unchanged | unchanged | unchanged | | Tool result | unchanged | `tool_result.content` blocks in order | as for a user message, same message | as for a user message | | Assistant | unchanged | raises, role reason (new wording) | role reason | role reason | #### Behavioral contracts 1. A tool result with media is sent as one `tool_result` block: `tool_use_id` is the call ID, `is_error` the result's flag, and `content` its blocks in order: `TextBlock` as `text`, `ImageBlock` as `image`, `DocumentBlock` as `document` titled by the block's `name`. 2. Media in a tool result follows the user-message rules: JPEG, PNG, GIF or WebP images by URL or Base64; PDF by URL or Base64; plain text by Base64, sent as decoded UTF-8. 3. Java and Python send the same JSON for a tool result with media. 4. A text-only tool result is sent as before: Java one concatenated string, Python a list of text blocks. 5. Audio, video, other image or document types, and plain text by URL in a tool result raise `UnsupportedContentBlockException` (Java) / `UnsupportedContentBlockError` (Python) with the same message as in a user message. The message names no URL or data. 6. Undecodable Base64 plain text in a tool result raises `IllegalArgumentException` / `ValueError`, not the unsupported type. 7. Media in an assistant message raises with the reason `only user messages and tool results can carry media, not assistant messages`. 8. Each tool message becomes its own `tool_result` block with its own call ID and content, text-only or not. #### Failure behavior Every failure above is raised while the request is built, so nothing is sent; media is never dropped or stringified. Java `chat()` rethrows `UnsupportedContentBlockException` unchanged and wraps other build errors in `RuntimeException`, as for user messages. Media the model rejects comes back as the service's error. ### Tests | Contract | Java `AnthropicMultimodalTest` / Python `test_anthropic_multimodal.py` | |---|---| | 1, 2 | `testToolResultMediaIsSentInOrder` / `test_tool_result_media_is_sent_in_order`; `testErrorToolResultWithMedia` / `test_error_tool_result_with_media` | | 3 | both order tests assert the same full `tool_result` | | 4 | `testTextOnlyToolResultIsUnchanged` (exact JSON) / `test_text_only_tool_result_is_unchanged` | | 5 | `testUnsupportedToolResultMediaFailsAsForUserMessages` / `test_unsupported_tool_result_media_fails_as_for_user_messages` | | 6 | `testInvalidBase64TextDocumentInToolResultFails` / `test_invalid_base64_text_document_in_tool_result_fails` | | 7 | `testMediaInAssistantMessagesFails` / `test_media_in_assistant_messages_fails` | | 8 | `testParallelToolResultsKeepTheirCorrelation` / `test_parallel_tool_results_keep_their_correlation` | | live | `test_anthropic_reads_image_returned_by_tool` (`test_anthropic_chat_model.py`): a tool returns a red PNG; the answer must say "red" | Coverage by risk: each accepted media kind and source appears in a tool result in both languages, and each unsupported kind is compared with its user-message error. Not verified: - The live test needs `TEST_API_KEY` (an Anthropic key). CI doesn't run it: no workflow sets that variable, and secrets aren't passed to fork PRs. It has not been run locally either. - No test sends a document, or an error result with media, in a tool result to the real API. - No end-to-end agent job returns media from a tool. <details> <summary>Test runs, test-file changes and implementation invariants</summary> - `mvn -B test -pl integrations/chat-models/anthropic`: 118 passed (`AnthropicMultimodalTest` 13). - `pytest -k "not e2e_tests" -m "not integration" flink_agents/integrations/chat_models/anthropic flink_agents/api`: 1201 passed, 15 skipped. Whole unit suite: 2877 passed, 16 skipped. - `./tools/lint.sh --check` (ruff 0.11.13, Spotless), `ruff format --check` and `./tools/check-license.sh` pass. - With the two adapter files reverted, the new tool-media, assistant-reason, unsupported, undecodable and parallel tests fail in both languages, and the text-only tests pass. - The former assistant-and-tool media test is split: the assistant half is contract 7's test; the tool half is replaced by the tests for contracts 1 and 5. The five unsupported blocks are a shared list used by the user and tool tests. - The live test calls the connection directly, with `TEST_MODEL` or the default model. - Java picks the text-only path with `allMatch(TextBlock)`, so an empty result is still sent as `""`. - `ToolResultBlock` holds only text and media blocks, so no other block type reaches `toToolResultContent`. - `ToolResponse.error(...)` builds a text-only result, so media with `is_error` comes only from a `ToolResultBlock` built directly. </details> ### API No signature or dependency change. For a caller who changes nothing: text-only tool results are sent unchanged; supported media in a tool result is now sent instead of raising; unsupported media still raises the same type, now with the user-message reason instead of `only user messages can carry media, not tool messages`. The assistant reason adds "and tool results". ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Claude Code 2.1.309 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
