Zhuoxi2000 opened a new pull request, #1326: URL: https://github.com/apache/flink-agents/pull/1326
<!-- * Thank you very much for contributing to Flink Agents. * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc. --> <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. --> Linked issue: #1059 ### Purpose of change <!-- What is the purpose of this change? --> A user `ChatMessage` with image, audio, video or document blocks now reaches Gemini with its media, replacing the rejection #1186 added. This is the Gemini part of the #1059 "Gemini, Bedrock, and DashScope" follow-up, after #1164 (OpenAI Chat Completions), #1178 (Ollama), #1204 (Anthropic) and #1275 (OpenAI Responses). The Gemini integration is Java-only, so there is no Python part. #### Runtime flow 1. `GeminiChatUtils.convertToContent` maps a user message's blocks in order to Gemini parts: each `TextBlock` as a text part, each media block as an `inlineData` part (Base64 data) or a `fileData` part (URL). 2. Media in an assistant message or a tool result throws. System messages cannot hold media since #1185. 3. A user message without media is sent exactly as before. #### Key decisions * All four media blocks map to the same two part shapes, since Gemini takes images, audio, video and documents through the same `Blob` / `FileData` parts. `mimeType` is the block's media type, which `MediaBlock` requires, so there is no fallback. * Media types and URLs are not restricted locally; Gemini rejects what a model or endpoint does not accept, as in #1164 and #1275. Which URLs the service can fetch differs between the Gemini Developer API and Vertex AI. * Base64 data is decoded with `java.util.Base64` for `Part.fromBytes`, as the Ollama and Anthropic integrations do; malformed data throws `IllegalArgumentException` while the request is built. * Tool results stay text-only: multimodal function responses are the separate "MCP image tool results" item of #1059. * The conversion runs inside `chat()`'s `try`, whose catch rethrows `IllegalArgumentException` unwrapped; `UnsupportedContentBlockException` extends it, so both reach callers with their own types. ### Behavioral Semantics <!-- For a non-trivial code change whose implementation is largely AI-assisted: interaction decisions, behavioral contracts, and failure behavior. See `contribution-guides/ai-assisted-pr.md`. Remove this heading and this comment otherwise. --> #### Interaction decisions | Role | Blocks | Result | |---|---|---| | user | text only | unchanged: one text part per `TextBlock` | | user | images, audio, video and documents, with or without text | parts in block order | | assistant, or a tool result | any media | `UnsupportedContentBlockException`; nothing sent | #### Behavioral contracts 1. A user message without media is sent exactly as before. 2. A user message with media is sent as parts in block order, one text part per `TextBlock`. 3. A media block with Base64 data becomes an `inlineData` part with the decoded bytes and `mimeType` = the block's media type. 4. A media block with a URL becomes a `fileData` part with `fileUri` = the URL and `mimeType` = the block's media type. 5. Media in an assistant message or a tool result throws `UnsupportedContentBlockException` naming the block but not its data or URL. 6. That exception, and the `IllegalArgumentException` for malformed Base64 data, reach callers of `chat()` with their own types. #### Failure behavior * Media in an assistant message or a tool result, or malformed Base64 data: thrown while building the request, so nothing is sent; the error is deterministic, so RETRY fails the same way. * A media type, URL or size Gemini rejects (an unsupported type, a URL the endpoint cannot fetch, an oversized request): the provider's error propagates as before. ### Tests <!-- How is this change verified? --> | Contract | `GeminiMultimodalTest` | |---|---| | 1 | `testTextOnlyUserMessageIsUnchanged` | | 2, 3 | `testBase64MediaBecomesInlineData` | | 2, 4 | `testUrlMediaBecomesFileData` | | 5 | `testMediaOutsideUserMessagesFails` | | 6 | `testMalformedBase64Fails`, `testChatKeepsMediaErrorTypes` | Tests assert on the SDK `Part` objects. #1186's `testMediaBlocksFailExplicitly` in `GeminiChatModelConnectionTest` is removed. Not verified: a live Gemini Developer API or Vertex AI endpoint, which needs credentials. <details> <summary>Implementation invariants and supporting evidence</summary> * google-genai 1.56.0: `Part.fromBytes(byte[], String)` builds an `inlineData` `Blob` and `Part.fromUri(String, String)` a `fileData` part, each with the given MIME type. * `UnsupportedContentBlockException.rejectMedia` stays: Bedrock and watsonx still use it. * Verified locally on main: the Gemini integration module 78/78, with spotless. </details> ### API <!-- Does this change touches any public APIs? --> No new API; it uses `UnsupportedContentBlockException` from #1164. Compatibility: text-only requests are unchanged. User media, rejected since #1186, is now sent. ### Documentation <!-- Do not remove this section. Check the proper box only. --> - [ ] `doc-needed` <!-- Your PR changes impact docs --> - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [x] `doc-included` <!-- Your PR already contains the necessary documentation updates --> A Gemini row in the Provider Capabilities table and a "Multimodal input" block table under Gemini in `providers.md`. ### Was this patch authored or co-authored using generative AI tooling? <!-- Do not remove this section. Check the proper box only. --> - [x] Yes - [ ] No If yes, include a `Generated-by: <tool name and version> (<model name and version>)` line, for example `Generated-by: Claude Code 2.1.226 (Claude Opus 4.6)`, in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html). Generated-by: Claude Code 2.1.259 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
