wenjin272 commented on code in PR #1370:
URL: https://github.com/apache/flink-agents/pull/1370#discussion_r4511483404
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -79,7 +79,7 @@ request is sent.
| [OpenAI Responses](#responses-api) | The media accepted in user messages,
sent as `function_call_output` output items in block order. The tool result's
error flag is not sent, because the API has no field for it. |
| [Anthropic](#anthropic) | The media accepted in user messages, sent in the
`tool_result` content in block order. |
| [Ollama](#ollama) | Not supported |
-| [Gemini](#gemini) | Not supported |
+| [Gemini](#gemini) | The media accepted in user messages, sent as the `parts`
of the `functionResponse` in block order, with the tool result's text in its
`response`. Google documents media in function responses for Gemini 3 models. |
Review Comment:
Could we simplify the **Media in tool results** documentation for Gemini,
Anthropic, and OpenAI Responses to focus on supported media types, similar to
**Media input accepted by the integration** above? The provider-specific
request structures and conversion details seem unnecessary for users.
##########
integrations/chat-models/gemini/src/main/java/org/apache/flink/agents/integrations/chatmodels/gemini/GeminiChatUtils.java:
##########
@@ -165,37 +167,37 @@ static Tool
convertTools(List<org.apache.flink.agents.api.tools.Tool> tools) {
}
/**
- * Converts a message to a Gemini turn. Only user messages can carry
media: image, audio, video
- * and document blocks become inline data (Base64) or file data (URL)
parts with the block's
- * media type. Media in an assistant message or a tool result throws {@link
- * UnsupportedContentBlockException}.
+ * Converts a message to a Gemini turn. Only user messages and tool
results can carry media:
+ * image, audio, video and document blocks become inline data (Base64) or
file data (URL) with
+ * the block's media type. A tool result's media become the parts of its
function response, in
+ * block order, and its text stays in the response's {@code result}. Media
in an assistant
+ * message throws {@link UnsupportedContentBlockException}.
*/
// Package-visible for unit testing of the message conversion.
static Content convertToContent(ChatMessage message, Map<String, String>
toolCallIdToName) {
if (message.getRole() == MessageRole.TOOL) {
ToolResultBlock result = (ToolResultBlock)
message.getBlocks().get(0);
- String text =
- result.getBlocks().stream()
- .map(
- block -> {
- if (block instanceof MediaBlock) {
- throw
mediaOutsideUserMessage(message, block);
- }
- if (!(block instanceof TextBlock)) {
- throw new IllegalArgumentException(
- "Gemini cannot send tool
result block "
- + block.getType());
- }
- return ((TextBlock) block).getText();
- })
- .collect(Collectors.joining());
+ List<FunctionResponsePart> media = new ArrayList<>();
+ for (DataContentBlock block : result.getBlocks()) {
+ if (block instanceof MediaBlock) {
Review Comment:
Could we restrict tool-result media to the supported MIME types
(PNG/JPEG/WebP, PDF, and plain text), and explicitly reject audio/video? Gemini
accepts audio/video in user messages, but rejects them in
`functionResponse.parts`. Gemini CLI encountered the same 400 errors and [added
filtering for these
types](https://github.com/google-gemini/gemini-cli/pull/26352). The tests and
documentation should reflect this distinction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
