This is an automated email from the ASF dual-hosted git repository.
wenjin272 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 29975213d [integrations][java][python][docs] Send images and documents
in tool results to Anthropic (#1331)
29975213d is described below
commit 29975213d294c90e57017f5fcb423452561fd9a3
Author: Shawn <[email protected]>
AuthorDate: Thu Oct 29 00:12:05 2026 -0700
[integrations][java][python][docs] Send images and documents in tool
results to Anthropic (#1331)
Generated-by: Claude Code 2.1.309 (Claude Opus 5.5)
---
.../calling-llms/chat-models/providers.md | 24 +-
.../docs/development/calling-llms/tool_use.md | 9 +-
.../chatmodels/anthropic/AnthropicChatUtils.java | 108 +++++----
.../anthropic/AnthropicMultimodalTest.java | 248 ++++++++++++++++++---
.../chat_models/anthropic/anthropic_chat_model.py | 11 +-
.../anthropic/tests/test_anthropic_chat_model.py | 62 +++++-
.../anthropic/tests/test_anthropic_multimodal.py | 235 ++++++++++++++++---
7 files changed, 580 insertions(+), 117 deletions(-)
diff --git
a/docs/content/docs/development/calling-llms/chat-models/providers.md
b/docs/content/docs/development/calling-llms/chat-models/providers.md
index fb12cb573..393cad898 100644
--- a/docs/content/docs/development/calling-llms/chat-models/providers.md
+++ b/docs/content/docs/development/calling-llms/chat-models/providers.md
@@ -61,12 +61,20 @@ model, deployment, and server must also support the
requested input and tool cal
All listed integrations can send tool definitions and return tool calls when
the
selected model supports them. Register the tools and reference them from the
setup;
see [Tool Use]({{< ref "docs/development/calling-llms/tool_use" >}}) for
declaration
-and execution behavior. Media support above applies to user messages, not to
tool
-results or other roles. Media that the integration does not support raises
-`UnsupportedContentBlockException` in Java or `UnsupportedContentBlockError` in
-Python before the request is sent. Media accepted by the integration can still
-be rejected by the model service if the selected model or deployment does not
-support it.
+and execution behavior. Media support above applies to user messages; tool
results
+are covered below, and other roles carry text only. Media that the integration
does
+not support raises `UnsupportedContentBlockException` in Java or
+`UnsupportedContentBlockError` in Python before the request is sent. Media
accepted
+by the integration can still be rejected by the model service if the selected
model
+or deployment does not support it.
+
+**Media in tool results.** Media returned by a tool reaches the model only
through
+the Anthropic integration:
+
+| Integration | Media in tool results |
+|---|---|
+| [Anthropic](#anthropic) | The media accepted in user messages, sent in the
`tool_result` content in block order. |
+| [Alibaba Cloud Model Studio
(DashScope)](#alibaba-cloud-model-studio-dashscope), [OpenAI Chat
Completions](#multimodal-input), [Azure OpenAI](#azure-openai), [vLLM](#vllm),
[OpenAI Responses](#responses-api), [Ollama](#ollama), [Gemini](#gemini),
[Amazon Bedrock](#amazon-bedrock), [IBM watsonx.ai](#watsonx-ibm-watsonxai) |
Not supported: raises `UnsupportedContentBlockException` (Java) or
`UnsupportedContentBlockError` (Python) before the request is sent. |
The implementation language does not limit the Agent API you can use: see
[Cross-Language Resources]({{< ref
"docs/development/create-agent/resources#cross-language-resources" >}})
@@ -364,7 +372,7 @@ public class MyAgent extends Agent {
{{< /tabs >}}
-**Multimodal input.** The media blocks of a user message are sent as Anthropic
content blocks, in block order, next to its text blocks:
+**Multimodal input.** The media blocks of a user message are sent as Anthropic
content blocks, in block order, next to its text blocks. The media blocks of a
tool result are sent the same way, with its text blocks, inside its
`tool_result` content:
| Block | Sent as |
|---|---|
@@ -373,7 +381,7 @@ public class MyAgent extends Agent {
| `DocumentBlock` | a `document` block titled by the block's `name`:
`application/pdf` by URL or Base64 data, or `text/plain` Base64 data sent as
decoded UTF-8 text |
| `AudioBlock`, `VideoBlock` | not supported |
-Other image or document types, plain text given by URL, audio, video, and
media in assistant messages or tool results raise
`UnsupportedContentBlockException` (Java) or `UnsupportedContentBlockError`
(Python) before the request is sent.
+Other image or document types, plain text given by URL, audio, video, and
media in assistant messages raise `UnsupportedContentBlockException` (Java) or
`UnsupportedContentBlockError` (Python) before the request is sent.
#### Available Models
diff --git a/docs/content/docs/development/calling-llms/tool_use.md
b/docs/content/docs/development/calling-llms/tool_use.md
index dde007f38..d8b33cd46 100644
--- a/docs/content/docs/development/calling-llms/tool_use.md
+++ b/docs/content/docs/development/calling-llms/tool_use.md
@@ -517,10 +517,11 @@ the model receives the error description as the tool
result.
### Representing media returned by tools
{{< hint warning >}}
-Current model integrations do not support sending media returned by tools to
the
-model. A tool response containing media causes the chat request to fail, even
if
-the model supports media in user messages. Use text-only tool responses in chat
-workflows for now.
+Only the Anthropic integration sends media returned by tools to the model.
With any
+other integration, a tool response containing media causes the chat request to
fail
+before it is sent, even if the model supports media in user messages. See
+[Provider Capabilities]({{< ref
"docs/development/calling-llms/chat-models/providers#provider-capabilities" >}})
+for the media each integration accepts.
{{< /hint >}}
Media must be provided explicitly as blocks. Returning bytes or a URL as an
diff --git
a/integrations/chat-models/anthropic/src/main/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicChatUtils.java
b/integrations/chat-models/anthropic/src/main/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicChatUtils.java
index db1f431cc..a857d49cb 100644
---
a/integrations/chat-models/anthropic/src/main/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicChatUtils.java
+++
b/integrations/chat-models/anthropic/src/main/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicChatUtils.java
@@ -43,6 +43,7 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.flink.agents.api.chat.messages.Base64Source;
import org.apache.flink.agents.api.chat.messages.ChatMessage;
import org.apache.flink.agents.api.chat.messages.ChatResult;
+import org.apache.flink.agents.api.chat.messages.DataContentBlock;
import org.apache.flink.agents.api.chat.messages.DocumentBlock;
import org.apache.flink.agents.api.chat.messages.ImageBlock;
import org.apache.flink.agents.api.chat.messages.MediaBlock;
@@ -81,6 +82,9 @@ final class AnthropicChatUtils {
private static final Set<String> IMAGE_MEDIA_TYPES =
Set.of("image/jpeg", "image/png", "image/gif", "image/webp");
+ private static final String UNSUPPORTED_MEDIA =
+ "Anthropic accepts images and PDF or plain-text documents only";
+
private AnthropicChatUtils() {}
static List<TextBlockParam> extractSystemMessages(List<ChatMessage>
messages) {
@@ -91,19 +95,18 @@ final class AnthropicChatUtils {
}
/**
- * Converts a message to an Anthropic message. Only user messages can
carry media: images (JPEG,
- * PNG, GIF or WebP) and PDF documents by Base64 data or URL, and
plain-text documents by Base64
- * data. Any other media, or media in an assistant message or a tool
result, throws {@link
+ * Converts a message to an Anthropic message. Only user messages and tool
results can carry
+ * media: images (JPEG, PNG, GIF or WebP) and PDF documents by Base64 data
or URL, and
+ * plain-text documents by Base64 data. A tool result's media goes into
its {@code tool_result}
+ * content. Any other media, or media in an assistant message, throws
{@link
* UnsupportedContentBlockException}.
*/
static MessageParam convertToAnthropicMessage(ChatMessage message) {
if (message.getRole() == MessageRole.TOOL) {
- ToolResultBlock result = (ToolResultBlock)
message.getBlocks().get(0);
- requireTextOnly(message, result.getBlocks());
- return convertToolResult(result);
+ return convertToolResult((ToolResultBlock)
message.getBlocks().get(0));
}
if (message.getRole() != MessageRole.USER) {
- requireTextOnly(message, message.getBlocks());
+ requireTextOnly(message);
}
List<ContentBlockParam> blocks = new ArrayList<>();
for (org.apache.flink.agents.api.chat.messages.ContentBlock block :
message.getBlocks()) {
@@ -145,29 +148,31 @@ final class AnthropicChatUtils {
private static ContentBlockParam toMediaBlockParam(MediaBlock block) {
if (block instanceof ImageBlock) {
- String mediaType = essence(block.getMediaType());
- if (!IMAGE_MEDIA_TYPES.contains(mediaType)) {
- throw unsupported(block, "images must be JPEG, PNG, GIF or
WebP");
- }
- ImageBlockParam.Builder param = ImageBlockParam.builder();
- if (block.getSource() instanceof UrlSource) {
- param.source(
- UrlImageSource.builder()
- .url(((UrlSource) block.getSource()).getUrl())
- .build());
- } else {
- param.source(
- Base64ImageSource.builder()
- .data(((Base64Source)
block.getSource()).getData())
-
.mediaType(Base64ImageSource.MediaType.of(mediaType))
- .build());
- }
- return ContentBlockParam.ofImage(param.build());
+ return ContentBlockParam.ofImage(toImageBlockParam((ImageBlock)
block));
}
if (block instanceof DocumentBlock) {
return
ContentBlockParam.ofDocument(toDocumentBlockParam((DocumentBlock) block));
}
- throw unsupported(block, "Anthropic accepts images and PDF or
plain-text documents only");
+ throw unsupported(block, UNSUPPORTED_MEDIA);
+ }
+
+ private static ImageBlockParam toImageBlockParam(ImageBlock image) {
+ String mediaType = essence(image.getMediaType());
+ if (!IMAGE_MEDIA_TYPES.contains(mediaType)) {
+ throw unsupported(image, "images must be JPEG, PNG, GIF or WebP");
+ }
+ ImageBlockParam.Builder param = ImageBlockParam.builder();
+ if (image.getSource() instanceof UrlSource) {
+ param.source(
+ UrlImageSource.builder().url(((UrlSource)
image.getSource()).getUrl()).build());
+ } else {
+ param.source(
+ Base64ImageSource.builder()
+ .data(((Base64Source) image.getSource()).getData())
+
.mediaType(Base64ImageSource.MediaType.of(mediaType))
+ .build());
+ }
+ return param.build();
}
private static DocumentBlockParam toDocumentBlockParam(DocumentBlock
document) {
@@ -207,14 +212,12 @@ final class AnthropicChatUtils {
throw unsupported(document, "documents must be PDF, or Base64 plain
text, for Anthropic");
}
- private static void requireTextOnly(
- ChatMessage message,
- List<? extends
org.apache.flink.agents.api.chat.messages.ContentBlock> blocks) {
- for (org.apache.flink.agents.api.chat.messages.ContentBlock block :
blocks) {
+ private static void requireTextOnly(ChatMessage message) {
+ for (org.apache.flink.agents.api.chat.messages.ContentBlock block :
message.getBlocks()) {
if (block instanceof MediaBlock) {
throw unsupported(
block,
- "only user messages can carry media, not "
+ "only user messages and tool results can carry media,
not "
+ message.getRole().getValue()
+ " messages");
}
@@ -231,28 +234,43 @@ final class AnthropicChatUtils {
return UnsupportedContentBlockException.forBlock("Anthropic", block,
reason);
}
+ /** Sends a text-only result as one string, and a result with media as
blocks in block order. */
private static MessageParam convertToolResult(ToolResultBlock result) {
- StringBuilder text = new StringBuilder();
- for (org.apache.flink.agents.api.chat.messages.ContentBlock block :
result.getBlocks()) {
- if (!(block instanceof TextBlock)) {
- throw new IllegalArgumentException(
- "Anthropic cannot send block type " + block.getType());
+ ToolResultBlockParam.Builder param =
+ ToolResultBlockParam.builder()
+ .toolUseId(result.getCallId())
+ .isError(result.isError());
+ if (result.getBlocks().stream().allMatch(block -> block instanceof
TextBlock)) {
+ param.content(result.getText());
+ } else {
+ List<ToolResultBlockParam.Content.Block> content = new
ArrayList<>();
+ for (DataContentBlock block : result.getBlocks()) {
+ content.add(toToolResultContent(block));
}
- text.append(((TextBlock) block).getText());
+ param.contentOfBlocks(content);
}
return MessageParam.builder()
.role(MessageParam.Role.USER)
- .contentOfBlockParams(
- List.of(
- ContentBlockParam.ofToolResult(
- ToolResultBlockParam.builder()
- .toolUseId(result.getCallId())
- .isError(result.isError())
- .content(text.toString())
- .build())))
+
.contentOfBlockParams(List.of(ContentBlockParam.ofToolResult(param.build())))
.build();
}
+ private static ToolResultBlockParam.Content.Block
toToolResultContent(DataContentBlock block) {
+ if (block instanceof TextBlock) {
+ return ToolResultBlockParam.Content.Block.ofText(
+ TextBlockParam.builder().text(((TextBlock)
block).getText()).build());
+ }
+ if (block instanceof ImageBlock) {
+ return ToolResultBlockParam.Content.Block.ofImage(
+ toImageBlockParam((ImageBlock) block));
+ }
+ if (block instanceof DocumentBlock) {
+ return ToolResultBlockParam.Content.Block.ofDocument(
+ toDocumentBlockParam((DocumentBlock) block));
+ }
+ throw unsupported(block, UNSUPPORTED_MEDIA);
+ }
+
/** Returns null for reasoning without this provider's native metadata. */
private static ContentBlockParam convertReasoning(ReasoningBlock
reasoning) {
Object nativeValue = reasoning.getMetadata().get(ANTHROPIC_METADATA);
diff --git
a/integrations/chat-models/anthropic/src/test/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicMultimodalTest.java
b/integrations/chat-models/anthropic/src/test/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicMultimodalTest.java
index 409a820a3..6a20d0ba3 100644
---
a/integrations/chat-models/anthropic/src/test/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicMultimodalTest.java
+++
b/integrations/chat-models/anthropic/src/test/java/org/apache/flink/agents/integrations/chatmodels/anthropic/AnthropicMultimodalTest.java
@@ -23,10 +23,12 @@ import com.fasterxml.jackson.databind.JsonNode;
import org.apache.flink.agents.api.chat.messages.AudioBlock;
import org.apache.flink.agents.api.chat.messages.Base64Source;
import org.apache.flink.agents.api.chat.messages.ChatMessage;
-import org.apache.flink.agents.api.chat.messages.ContentBlock;
+import org.apache.flink.agents.api.chat.messages.DataContentBlock;
import org.apache.flink.agents.api.chat.messages.DocumentBlock;
import org.apache.flink.agents.api.chat.messages.ImageBlock;
+import org.apache.flink.agents.api.chat.messages.MediaBlock;
import org.apache.flink.agents.api.chat.messages.TextBlock;
+import org.apache.flink.agents.api.chat.messages.ToolCallBlock;
import org.apache.flink.agents.api.chat.messages.ToolResultBlock;
import
org.apache.flink.agents.api.chat.messages.UnsupportedContentBlockException;
import org.apache.flink.agents.api.chat.messages.VideoBlock;
@@ -41,6 +43,7 @@ import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
+import static org.assertj.core.api.Assertions.catchThrowable;
/** Tests how content blocks become Anthropic Messages API content blocks. */
class AnthropicMultimodalTest {
@@ -48,6 +51,15 @@ class AnthropicMultimodalTest {
private static final String URL = "https://example.com/cat.png?sig=secret";;
private static final String BASE64 = "aGVsbG8="; // "hello"
+ /** Media Anthropic has no block for. */
+ private static final List<MediaBlock> UNSUPPORTED =
+ List.of(
+ AudioBlock.fromBase64("audio/wav", BASE64),
+ VideoBlock.fromUrl("video/mp4", URL),
+ ImageBlock.fromBase64("image/bmp", BASE64),
+ DocumentBlock.fromUrl("text/plain", URL),
+ DocumentBlock.fromBase64("application/msword", BASE64));
+
private static AnthropicChatModelConnection connection() {
ResourceDescriptor descriptor =
ResourceDescriptor.Builder.newBuilder(AnthropicChatModelConnection.class.getName())
@@ -66,13 +78,29 @@ class AnthropicMultimodalTest {
/** The first request message, as the SDK serializes it. */
private static JsonNode wire(ChatMessage message) {
+ return wire(List.of(message)).get(0);
+ }
+
+ /** The request messages, as the SDK serializes them. */
+ private static JsonNode wire(List<ChatMessage> messages) {
return ObjectMappers.jsonMapper()
.valueToTree(
connection()
- .buildRequest(List.of(message), List.of(),
params(), null)
+ .buildRequest(messages, List.of(), params(),
null)
.params
- .messages()
- .get(0));
+ .messages());
+ }
+
+ private static JsonNode json(Object value) {
+ return ObjectMappers.jsonMapper().valueToTree(value);
+ }
+
+ /** The tool_result block a tool message is sent as. */
+ private static JsonNode toolResult(ToolResultBlock result) {
+ JsonNode message = wire(ChatMessage.tool(result));
+ assertThat(message.get("role").asText()).isEqualTo("user");
+ assertThat(message.get("content")).hasSize(1);
+ return message.at("/content/0");
}
@Test
@@ -139,14 +167,7 @@ class AnthropicMultimodalTest {
@Test
@DisplayName("Blocks Anthropic has no block for fail without leaking the
source")
void testUnsupportedBlocksFailExplicitly() {
- List<ContentBlock> unsupported =
- List.of(
- AudioBlock.fromBase64("audio/wav", BASE64),
- VideoBlock.fromUrl("video/mp4", URL),
- ImageBlock.fromBase64("image/bmp", BASE64),
- DocumentBlock.fromUrl("text/plain", URL),
- DocumentBlock.fromBase64("application/msword",
BASE64));
- for (ContentBlock block : unsupported) {
+ for (MediaBlock block : UNSUPPORTED) {
assertThatThrownBy(() ->
wire(ChatMessage.user(List.of(TextBlock.of("hi"), block))))
.isInstanceOf(UnsupportedContentBlockException.class)
.hasMessageStartingWith("Anthropic cannot send a")
@@ -157,32 +178,201 @@ class AnthropicMultimodalTest {
}
@Test
- @DisplayName("Media in assistant messages and tool results fails
explicitly")
- void testMediaOutsideUserMessagesFails() {
- List<ContentBlock> blocks =
- List.of(TextBlock.of("see"), ImageBlock.fromUrl("image/png",
URL));
- ChatMessage assistant = ChatMessage.assistant(blocks);
- ChatMessage tool =
- ChatMessage.tool(
+ @DisplayName("Media in assistant messages fails explicitly")
+ void testMediaInAssistantMessagesFails() {
+ ChatMessage assistant =
+ ChatMessage.assistant(
+ List.of(TextBlock.of("see"),
ImageBlock.fromUrl("image/png", URL)));
+
+ assertThatThrownBy(() -> wire(List.of(ChatMessage.user("hi"),
assistant)))
+ .isInstanceOf(UnsupportedContentBlockException.class)
+ .hasMessage(
+ "Anthropic cannot send an image block (image/png, url
source): only user"
+ + " messages and tool results can carry media,
not assistant"
+ + " messages.");
+ }
+
+ @Test
+ @DisplayName("A text-only tool result is sent as before, as one string")
+ void testTextOnlyToolResultIsUnchanged() {
+ JsonNode message =
+ wire(
+ ChatMessage.tool(
+ new ToolResultBlock(
+ "call-1",
+ List.of(TextBlock.of("4"),
TextBlock.of("2")),
+ false)));
+
+ assertThat(message.toString())
+ .isEqualTo(
+
"{\"content\":[{\"tool_use_id\":\"call-1\",\"type\":\"tool_result\","
+ +
"\"content\":\"42\",\"is_error\":false}],\"role\":\"user\"}");
+ }
+
+ @Test
+ @DisplayName("Media in a tool result goes into tool_result content, in
block order")
+ void testToolResultMediaIsSentInOrder() {
+ List<DataContentBlock> blocks =
+ List.of(
+ TextBlock.of("Chart:"),
+ ImageBlock.fromUrl("image/png", URL),
+ ImageBlock.fromBase64("image/png", BASE64),
+ new DocumentBlock(
+ "application/pdf",
+ new Base64Source(BASE64),
+ "report.pdf",
+ null,
+ null),
+ DocumentBlock.fromUrl("application/pdf",
"https://example.com/a.pdf";),
+ DocumentBlock.fromBase64("text/plain", BASE64),
+ TextBlock.of("Done."));
+
+ JsonNode result = toolResult(new ToolResultBlock("call-1", blocks,
false));
+
+ assertThat(result)
+ .isEqualTo(
+ json(
+ Map.of(
+ "type",
+ "tool_result",
+ "tool_use_id",
+ "call-1",
+ "content",
+ List.of(
+ Map.of("type", "text", "text",
"Chart:"),
+ Map.of(
+ "type",
+ "image",
+ "source",
+ Map.of("type", "url",
"url", URL)),
+ Map.of(
+ "type",
+ "image",
+ "source",
+ Map.of(
+ "type",
"base64",
+ "media_type",
"image/png",
+ "data",
BASE64)),
+ Map.of(
+ "type", "document",
+ "source",
+ Map.of(
+
"type", "base64",
+
"media_type",
+
"application/pdf",
+
"data", BASE64),
+ "title", "report.pdf"),
+ Map.of(
+ "type",
+ "document",
+ "source",
+ Map.of(
+ "type", "url",
+ "url",
+
"https://example.com/a.pdf";)),
+ Map.of(
+ "type",
+ "document",
+ "source",
+ Map.of(
+ "type", "text",
+ "media_type",
"text/plain",
+ "data",
"hello")),
+ Map.of("type", "text", "text",
"Done.")),
+ "is_error",
+ false)));
+ // Each block is sent exactly as in a user message.
+
assertThat(result.get("content")).isEqualTo(wire(ChatMessage.user(blocks)).get("content"));
+ }
+
+ @Test
+ @DisplayName("An error tool result with media keeps is_error")
+ void testErrorToolResultWithMedia() {
+ JsonNode result =
+ toolResult(
new ToolResultBlock(
"call-1",
- List.of(TextBlock.of("see"),
ImageBlock.fromUrl("image/png", URL)),
- false));
+ List.of(
+ TextBlock.of("Render failed; partial
output:"),
+ ImageBlock.fromUrl("image/png", URL)),
+ true));
+
+ assertThat(result.get("tool_use_id").asText()).isEqualTo("call-1");
+ assertThat(result.get("is_error").asBoolean()).isTrue();
+ assertThat(result.at("/content/0/type").asText()).isEqualTo("text");
+ assertThat(result.at("/content/1/type").asText()).isEqualTo("image");
+ }
+
+ @Test
+ @DisplayName("Media Anthropic cannot send fails in a tool result as in a
user message")
+ void testUnsupportedToolResultMediaFailsAsForUserMessages() {
+ for (MediaBlock block : UNSUPPORTED) {
+ Throwable inUserMessage = catchThrowable(() ->
wire(ChatMessage.user(List.of(block))));
+
assertThat(inUserMessage).isInstanceOf(UnsupportedContentBlockException.class);
- for (ChatMessage message : List.of(assistant, tool)) {
assertThatThrownBy(
() ->
- connection()
- .buildRequest(
-
List.of(ChatMessage.user("hi"), message),
- List.of(),
- params(),
- null))
+ wire(
+ ChatMessage.tool(
+ new ToolResultBlock(
+ "call-1",
+
List.of(TextBlock.of("hi"), block),
+ false))))
.isInstanceOf(UnsupportedContentBlockException.class)
- .hasMessageContaining("only user messages can carry
media");
+ .hasMessage(inUserMessage.getMessage())
+ .hasMessageNotContaining("secret")
+ .hasMessageNotContaining(BASE64);
}
}
+ @Test
+ @DisplayName("Undecodable plain-text data in a tool result fails as
invalid")
+ void testInvalidBase64TextDocumentInToolResultFails() {
+ ChatMessage message =
+ ChatMessage.tool(
+ new ToolResultBlock(
+ "call-1",
+ List.of(DocumentBlock.fromBase64("text/plain",
"not base64!")),
+ false));
+
+ assertThatThrownBy(() -> wire(message))
+ .isExactlyInstanceOf(IllegalArgumentException.class)
+ .hasMessage("A document block's base64 data could not be
decoded.");
+ }
+
+ @Test
+ @DisplayName("Parallel tool results keep their own call IDs and content")
+ void testParallelToolResultsKeepTheirCorrelation() {
+ JsonNode messages =
+ wire(
+ List.of(
+ ChatMessage.user("What is the weather, and
show me the map."),
+ ChatMessage.assistant(
+ List.of(
+ new ToolCallBlock("call-1",
"weather", Map.of()),
+ new ToolCallBlock("call-2",
"map", Map.of()))),
+ ChatMessage.tool(
+ new ToolResultBlock(
+ "call-1",
List.of(TextBlock.of("Sunny")), false)),
+ ChatMessage.tool(
+ new ToolResultBlock(
+ "call-2",
+ List.of(
+ TextBlock.of("Map:"),
+
ImageBlock.fromUrl("image/png", URL)),
+ false))));
+
+ assertThat(messages).hasSize(4);
+ assertThat(messages.at("/2/content")).hasSize(1);
+
assertThat(messages.at("/2/content/0/tool_use_id").asText()).isEqualTo("call-1");
+
assertThat(messages.at("/2/content/0/content").asText()).isEqualTo("Sunny");
+ assertThat(messages.at("/3/content")).hasSize(1);
+
assertThat(messages.at("/3/content/0/tool_use_id").asText()).isEqualTo("call-2");
+ assertThat(messages.at("/3/content/0/content")).hasSize(2);
+
assertThat(messages.at("/3/content/0/content/0/text").asText()).isEqualTo("Map:");
+
assertThat(messages.at("/3/content/0/content/1/source/url").asText()).isEqualTo(URL);
+ }
+
@Test
@DisplayName("Undecodable plain-text data fails as invalid, not as
unsupported")
void testInvalidBase64TextDocumentFails() {
diff --git
a/python/flink_agents/integrations/chat_models/anthropic/anthropic_chat_model.py
b/python/flink_agents/integrations/chat_models/anthropic/anthropic_chat_model.py
index 417dfc414..0cb5d1b19 100644
---
a/python/flink_agents/integrations/chat_models/anthropic/anthropic_chat_model.py
+++
b/python/flink_agents/integrations/chat_models/anthropic/anthropic_chat_model.py
@@ -145,9 +145,10 @@ def _to_media_block(block: MediaBlock) -> Dict[str, Any]:
def convert_to_anthropic_message(message: ChatMessage) -> MessageParam:
"""Convert ChatMessage to Anthropic MessageParam format.
- Only user messages can carry media: images (JPEG, PNG, GIF or WebP) and PDF
- documents by Base64 data or URL, and plain-text documents by Base64 data.
- Any other media, or media in an assistant message or a tool result, raises
+ Only user messages and tool results can carry media: images (JPEG, PNG, GIF
+ or WebP) and PDF documents by Base64 data or URL, and plain-text documents
by
+ Base64 data. A tool result's media goes into its tool_result content. Any
+ other media, or media in an assistant message, raises
UnsupportedContentBlockError.
"""
blocks = (
@@ -158,9 +159,9 @@ def convert_to_anthropic_message(message: ChatMessage) ->
MessageParam:
if isinstance(block, TextBlock):
content.append({"type": "text", "text": block.text})
elif isinstance(block, MediaBlock):
- if message.role != MessageRole.USER:
+ if message.role not in (MessageRole.USER, MessageRole.TOOL):
reason = (
- "only user messages can carry media, "
+ "only user messages and tool results can carry media, "
f"not {message.role.value} messages"
)
raise _unsupported(block, reason)
diff --git
a/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_chat_model.py
b/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_chat_model.py
index c8332bc7a..632683a1f 100644
---
a/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_chat_model.py
+++
b/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_chat_model.py
@@ -15,19 +15,31 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#################################################################################
+import base64
import os
+import struct
+import zlib
from unittest.mock import MagicMock
import pytest
-from flink_agents.api.chat_message import ChatMessage, MessageRole
+from flink_agents.api.chat_message import (
+ ChatMessage,
+ ImageBlock,
+ MessageRole,
+ TextBlock,
+ ToolResultBlock,
+)
from flink_agents.api.resource import Resource, ResourceType
from flink_agents.api.resource_context import ResourceContext
from flink_agents.api.tools.tool import Tool
from flink_agents.integrations.chat_models.anthropic.anthropic_chat_model
import (
+ DEFAULT_ANTHROPIC_MODEL,
AnthropicChatModelConnection,
AnthropicChatModelSetup,
)
+from flink_agents.plan.function import PythonFunction
+from flink_agents.plan.tools.function_tool import FunctionTool
pytestmark = pytest.mark.integration
@@ -103,6 +115,54 @@ def test_anthropic_chat_with_tools() -> None:
assert tool_call.call_id is not None
+def get_swatch() -> str:
+ """Return the color swatch to identify, as an image."""
+ return "The test returns the swatch image itself."
+
+
+def _red_square_png() -> str:
+ """A 16x16 red PNG."""
+ rows = b"".join(b"\x00" + b"\xff\x00\x00" * 16 for _ in range(16))
+
+ def chunk(kind: bytes, data: bytes) -> bytes:
+ body = kind + data
+ return struct.pack(">I", len(data)) + body + struct.pack(">I",
zlib.crc32(body))
+
+ png = (
+ b"\x89PNG\r\n\x1a\n"
+ + chunk(b"IHDR", struct.pack(">IIBBBBB", 16, 16, 8, 2, 0, 0, 0))
+ + chunk(b"IDAT", zlib.compress(rows))
+ + chunk(b"IEND", b"")
+ )
+ return base64.b64encode(png).decode()
+
+
[email protected](api_key is None, reason="TEST_API_KEY is not set")
+def test_anthropic_reads_image_returned_by_tool() -> None:
+ """The model calls a tool, gets an image in the tool result and uses it."""
+ connection = AnthropicChatModelConnection(api_key=api_key)
+ tools = [FunctionTool(func=PythonFunction.from_callable(get_swatch))]
+ params = {"model": test_model or DEFAULT_ANTHROPIC_MODEL, "max_tokens":
256}
+ question = ChatMessage.user(
+ "Call get_swatch, then answer with the color of the swatch in one
word."
+ )
+
+ first = connection.chat([question], tools=tools, **params)
+ assert len(first.tool_calls) == 1
+
+ swatch = ToolResultBlock(
+ call_id=first.tool_calls[0].call_id,
+ blocks=(
+ TextBlock(text="The swatch:"),
+ ImageBlock.from_base64("image/png", _red_square_png()),
+ ),
+ )
+ second = connection.chat(
+ [question, first.message, ChatMessage.tool(swatch)], tools=tools,
**params
+ )
+ assert "red" in second.text.lower()
+
+
def test_model_field_roundtrip() -> None:
"""Verify `model` is preserved through pydantic dump/validate
round-trip."""
setup = AnthropicChatModelSetup(connection="conn", model="test-model")
diff --git
a/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_multimodal.py
b/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_multimodal.py
index 9956de1f1..2ebbe7b79 100644
---
a/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_multimodal.py
+++
b/python/flink_agents/integrations/chat_models/anthropic/tests/test_anthropic_multimodal.py
@@ -20,7 +20,7 @@
Mirrors the Java AnthropicMultimodalTest.
"""
-from typing import Any, Dict
+from typing import Any, Dict, List
from unittest.mock import MagicMock
import pytest
@@ -35,6 +35,7 @@ from flink_agents.api.chat_message import (
DocumentBlock,
ImageBlock,
TextBlock,
+ ToolCallBlock,
ToolResultBlock,
UnsupportedContentBlockError,
VideoBlock,
@@ -46,6 +47,15 @@ from
flink_agents.integrations.chat_models.anthropic.anthropic_chat_model import
URL = "https://example.com/cat.png?sig=secret";
BASE64 = "aGVsbG8=" # "hello"
+# Media Anthropic has no block for.
+UNSUPPORTED = [
+ AudioBlock.from_base64("audio/wav", BASE64),
+ VideoBlock.from_url("video/mp4", URL),
+ ImageBlock.from_base64("image/bmp", BASE64),
+ DocumentBlock.from_url("text/plain", URL),
+ DocumentBlock.from_base64("application/msword", BASE64),
+]
+
def _connection() -> tuple[AnthropicChatModelConnection, MagicMock]:
connection = AnthropicChatModelConnection(api_key="fake-key")
@@ -71,7 +81,20 @@ def _chat(*messages: ChatMessage) -> MagicMock:
def _sent(message: ChatMessage) -> Dict[str, Any]:
"""Chat through a mocked client and return the first message as sent."""
- return _chat(message).messages.create.call_args.kwargs["messages"][0]
+ return _sent_all(message)[0]
+
+
+def _sent_all(*messages: ChatMessage) -> List[Dict[str, Any]]:
+ """Chat through a mocked client and return the messages as sent."""
+ return _chat(*messages).messages.create.call_args.kwargs["messages"]
+
+
+def _tool_result(result: ToolResultBlock) -> Dict[str, Any]:
+ """The tool_result block a tool message is sent as."""
+ message = _sent(ChatMessage.tool(result))
+ assert message["role"] == "user"
+ assert len(message["content"]) == 1
+ return message["content"][0]
def test_text_only_user_message_is_unchanged() -> None:
@@ -140,17 +163,7 @@ def test_documents_become_document_blocks() -> None:
]
[email protected](
- "block",
- [
- AudioBlock.from_base64("audio/wav", BASE64),
- VideoBlock.from_url("video/mp4", URL),
- ImageBlock.from_base64("image/bmp", BASE64),
- DocumentBlock.from_url("text/plain", URL),
- DocumentBlock.from_base64("application/msword", BASE64),
- ],
- ids=str,
-)
[email protected]("block", UNSUPPORTED, ids=str)
def test_unsupported_blocks_fail_explicitly(block: ContentBlock) -> None:
"""Blocks Anthropic has no block for fail without leaking the source."""
connection, client = _connection()
@@ -167,20 +180,192 @@ def test_unsupported_blocks_fail_explicitly(block:
ContentBlock) -> None:
client.messages.create.assert_not_called()
[email protected]("kind", ["assistant", "tool"])
-def test_media_outside_user_messages_fails(kind: str) -> None:
- """Media in assistant messages and tool results fails explicitly."""
- blocks = [TextBlock(text="see"), ImageBlock.from_url("image/png", URL)]
- message = (
- ChatMessage.assistant(blocks)
- if kind == "assistant"
- else ChatMessage.tool(ToolResultBlock(call_id="call-1",
blocks=tuple(blocks)))
+def test_media_in_assistant_messages_fails() -> None:
+ """Media in assistant messages fails explicitly."""
+ assistant = ChatMessage.assistant(
+ [TextBlock(text="see"), ImageBlock.from_url("image/png", URL)]
+ )
+
+ with pytest.raises(UnsupportedContentBlockError) as error:
+ _chat(ChatMessage.user("hi"), assistant)
+
+ assert str(error.value) == (
+ "Anthropic cannot send an image block (image/png, url source): only
user "
+ "messages and tool results can carry media, not assistant messages."
+ )
+
+
+def test_text_only_tool_result_is_unchanged() -> None:
+ """A text-only tool result is sent as before, as a list of text blocks."""
+ result = ToolResultBlock(
+ call_id="call-1", blocks=(TextBlock(text="4"), TextBlock(text="2"))
+ )
+
+ assert _sent(ChatMessage.tool(result)) == {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call-1",
+ "content": [
+ {"type": "text", "text": "4"},
+ {"type": "text", "text": "2"},
+ ],
+ "is_error": False,
+ }
+ ],
+ }
+
+
+def test_tool_result_media_is_sent_in_order() -> None:
+ """Media in a tool result goes into tool_result content, in block order."""
+ blocks = (
+ TextBlock(text="Chart:"),
+ ImageBlock.from_url("image/png", URL),
+ ImageBlock.from_base64("image/png", BASE64),
+ DocumentBlock(
+ media_type="application/pdf",
+ source=Base64Source(data=BASE64),
+ name="report.pdf",
+ ),
+ DocumentBlock.from_url("application/pdf", "https://example.com/a.pdf";),
+ DocumentBlock.from_base64("text/plain", BASE64),
+ TextBlock(text="Done."),
)
- with pytest.raises(
- UnsupportedContentBlockError, match="only user messages can carry
media"
- ):
- _chat(ChatMessage.user("hi"), message)
+ result = _tool_result(ToolResultBlock(call_id="call-1", blocks=blocks))
+
+ assert result == {
+ "type": "tool_result",
+ "tool_use_id": "call-1",
+ "content": [
+ {"type": "text", "text": "Chart:"},
+ {"type": "image", "source": {"type": "url", "url": URL}},
+ {
+ "type": "image",
+ "source": {"type": "base64", "media_type": "image/png",
"data": BASE64},
+ },
+ {
+ "type": "document",
+ "source": {
+ "type": "base64",
+ "media_type": "application/pdf",
+ "data": BASE64,
+ },
+ "title": "report.pdf",
+ },
+ {
+ "type": "document",
+ "source": {"type": "url", "url": "https://example.com/a.pdf"},
+ },
+ {
+ "type": "document",
+ "source": {"type": "text", "media_type": "text/plain", "data":
"hello"},
+ },
+ {"type": "text", "text": "Done."},
+ ],
+ "is_error": False,
+ }
+ # Each block is sent exactly as in a user message.
+ assert result["content"] ==
_sent(ChatMessage.user(list(blocks)))["content"]
+
+
+def test_error_tool_result_with_media() -> None:
+ """An error tool result with media keeps is_error."""
+ result = _tool_result(
+ ToolResultBlock(
+ call_id="call-1",
+ blocks=(
+ TextBlock(text="Render failed; partial output:"),
+ ImageBlock.from_url("image/png", URL),
+ ),
+ is_error=True,
+ )
+ )
+
+ assert result["tool_use_id"] == "call-1"
+ assert result["is_error"] is True
+ assert [block["type"] for block in result["content"]] == ["text", "image"]
+
+
[email protected]("block", UNSUPPORTED, ids=str)
+def test_unsupported_tool_result_media_fails_as_for_user_messages(
+ block: ContentBlock,
+) -> None:
+ """Media Anthropic cannot send fails in a tool result as in a user
message."""
+ with pytest.raises(UnsupportedContentBlockError) as in_user_message:
+ _chat(ChatMessage.user([block]))
+ connection, client = _connection()
+ message = ChatMessage.tool(
+ ToolResultBlock(call_id="call-1", blocks=(TextBlock(text="hi"), block))
+ )
+
+ with pytest.raises(UnsupportedContentBlockError) as error:
+ connection.chat([message], model="claude-sonnet-4-5")
+
+ text = str(error.value)
+ assert text == str(in_user_message.value)
+ assert "secret" not in text
+ assert BASE64 not in text
+ client.messages.create.assert_not_called()
+
+
+def test_invalid_base64_text_document_in_tool_result_fails() -> None:
+ """Undecodable plain-text data in a tool result fails with ValueError."""
+ message = ChatMessage.tool(
+ ToolResultBlock(
+ call_id="call-1",
+ blocks=(DocumentBlock.from_base64("text/plain", "not base64!"),),
+ )
+ )
+
+ with pytest.raises(ValueError, match="could not be decoded") as error:
+ _chat(message)
+
+ assert not isinstance(error.value, UnsupportedContentBlockError)
+
+
+def test_parallel_tool_results_keep_their_correlation() -> None:
+ """Parallel tool results keep their own call IDs and content."""
+ messages = _sent_all(
+ ChatMessage.user("What is the weather, and show me the map."),
+ ChatMessage.assistant(
+ [
+ ToolCallBlock(call_id="call-1", name="weather"),
+ ToolCallBlock(call_id="call-2", name="map"),
+ ]
+ ),
+ ChatMessage.tool(
+ ToolResultBlock(call_id="call-1",
blocks=(TextBlock(text="Sunny"),))
+ ),
+ ChatMessage.tool(
+ ToolResultBlock(
+ call_id="call-2",
+ blocks=(TextBlock(text="Map:"),
ImageBlock.from_url("image/png", URL)),
+ )
+ ),
+ )
+
+ assert len(messages) == 4
+ assert messages[2]["content"] == [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call-1",
+ "content": [{"type": "text", "text": "Sunny"}],
+ "is_error": False,
+ }
+ ]
+ assert messages[3]["content"] == [
+ {
+ "type": "tool_result",
+ "tool_use_id": "call-2",
+ "content": [
+ {"type": "text", "text": "Map:"},
+ {"type": "image", "source": {"type": "url", "url": URL}},
+ ],
+ "is_error": False,
+ }
+ ]
def test_invalid_base64_text_document_fails() -> None:
![https://example.com/cat.png?sig=secret"](https://example.com/cat.png?sig=secret"){kind=link}