Zhuoxi2000 opened a new pull request, #1367: URL: https://github.com/apache/flink-agents/pull/1367
<!-- * Thank you very much for contributing to Flink Agents. * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc. --> <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. --> Linked issue: #1059 ### Purpose of change <!-- What is the purpose of this change? --> A user `ChatMessage` with image, document, video or audio blocks now reaches Amazon Bedrock with its media, replacing the rejection #1186 added. This is the Bedrock part of the #1059 "Gemini, Bedrock, and DashScope" follow-up, after #1326 (Gemini). The Bedrock chat integration is Java-only, so there is no Python part. #### Runtime flow 1. `BedrockChatUtils` maps a user message's blocks in order to Converse content blocks: each `TextBlock` as a `text` block, each media block as an `image`, `document`, `video` or `audio` block. 2. The media type is mapped to the Converse `format`; the source becomes `bytes` (Base64 data) or an `s3Location` (an `s3://` URI). 3. Media in an assistant message or a tool result throws. System messages cannot hold media since #1185. 4. A user message without media is sent exactly as before. #### Key decisions * **Formats, not media types.** Converse takes a `format` enum, so media types map through one explicit table per block type (below), ignoring case and parameters such as `charset`. A media type without a Converse format throws rather than being guessed. * **Only `s3://` URLs.** Converse sources are `bytes` or `s3Location`; it cannot fetch HTTP URLs. Unlike #1326, any other URL throws. * **Document names.** Converse requires a `name` made only of ASCII letters and digits, single spaces, hyphens, parentheses and square brackets (1 to 200 characters). Other characters in the block's `name` become spaces (`Q3 report.pdf` becomes `Q3 report pdf`); an unnamed or emptied name becomes `document-<n>`, numbered within the message. * **Audio is included.** The SDK and API have `AudioBlock`; which models accept it, or S3 sources, varies, and the service rejects what a model does not accept, as in #1164 and #1326. * Base64 data is decoded with `java.util.Base64`, as the Gemini, Ollama and Anthropic integrations do; malformed data throws `IllegalArgumentException` while the request is built. * Tool results stay text-only here; media in tool results is #1216. * The request is built before the retried `converse` call and outside any `try`, so both exceptions reach callers with their own types. | Block | Media types | Converse format | |---|---|---| | `ImageBlock` | `image/png`, `image/jpeg`, `image/gif`, `image/webp` | `png`, `jpeg`, `gif`, `webp` | | `DocumentBlock` | `application/pdf`, `text/csv`, `application/msword`, `application/vnd.openxmlformats-officedocument.wordprocessingml.document`, `application/vnd.ms-excel`, `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`, `text/html`, `text/plain`, `text/markdown` | `pdf`, `csv`, `doc`, `docx`, `xls`, `xlsx`, `html`, `txt`, `md` | | `VideoBlock` | `video/x-matroska`, `video/quicktime`, `video/mp4`, `video/webm`, `video/x-flv`, `video/mpeg`, `video/x-ms-wmv`, `video/3gpp` | `mkv`, `mov`, `mp4`, `webm`, `flv`, `mpeg`, `wmv`, `three_gp` | | `AudioBlock` | `audio/mpeg`, `audio/opus`, `audio/wav`, `audio/x-wav`, `audio/aac`, `audio/x-aac`, `audio/flac`, `audio/mp4`, `audio/x-m4a`, `audio/ogg`, `audio/x-matroska`, `audio/webm` | `mp3`, `opus`, `wav`, `wav`, `aac`, `x-aac`, `flac`, `mp4`, `m4a`, `ogg`, `mka`, `webm` | ### Behavioral Semantics <!-- For a non-trivial code change whose implementation is largely AI-assisted: interaction decisions, behavioral contracts, and failure behavior. See `contribution-guides/ai-assisted-pr.md`. Remove this heading and this comment otherwise. --> #### Interaction decisions | Role | Blocks | Result | |---|---|---| | user | text only | unchanged: one `text` block per `TextBlock` | | user | media with a mapped type, by Base64 data or `s3://` URI | Converse blocks in block order | | user | media with an unmapped type, or another URL | `UnsupportedContentBlockException`; nothing sent | | assistant, or a tool result | any media | `UnsupportedContentBlockException`; nothing sent | #### Behavioral contracts 1. A user message without media is sent exactly as before. 2. A user message with media is sent as Converse blocks in block order, one `text` block per `TextBlock`. 3. Each media block becomes the Converse block of its type, with `format` from the table above and `bytes` (decoded Base64 data) or `s3Location.uri` (an `s3://` URI) as its source. 4. A `document` block's `name` is the block's `name` with disallowed characters replaced by spaces and capped at 200 characters, else `document-<n>`. 5. Unmapped media types, non-`s3://` URLs, and media in an assistant message or a tool result throw `UnsupportedContentBlockException` naming the block but not its data or URL. 6. That exception, and the `IllegalArgumentException` for malformed Base64 data, reach callers of `chat()` with their own types. #### Failure behavior * Unsupported media, URL or role, or malformed Base64 data: thrown while building the request, so nothing is sent; the error is deterministic, so RETRY fails the same way. * Media a model does not accept, an S3 object it cannot read, or an oversized request: the provider's error propagates as before. ### Tests <!-- How is this change verified? --> | Contract | `BedrockMultimodalTest` | |---|---| | 1 | `testTextOnlyUserMessageIsUnchanged` | | 2, 3 | `testBase64MediaBecomesBytes`, `testS3UrlsBecomeS3Locations`, `testMediaTypeParametersAreIgnored` | | 4 | `testDocumentNames` | | 5 | `testUnsupportedSourcesAndTypesFail`, `testMediaOutsideUserMessagesFails` | | 6 | `testChatKeepsMediaErrorTypes` | Tests assert on the SDK `ContentBlock` objects. #1186's `testMediaBlocksFailExplicitly` in `BedrockChatModelConnectionTest` is removed. Not verified: a live Bedrock endpoint, which needs AWS credentials. <details> <summary>Implementation invariants and supporting evidence</summary> * AWS SDK 2.41.22 `bedrockruntime`: `ContentBlock.fromImage` / `fromDocument` / `fromVideo` / `fromAudio`; each source has `bytes` and `s3Location`; formats are the `ImageFormat`, `DocumentFormat`, `VideoFormat` and `AudioFormat` enums, matching the API reference. * The `DocumentBlock.name` rule and the `s3://` pattern of `S3Location.uri` are from the Bedrock API reference. * `UnsupportedContentBlockException.rejectMedia` stays: watsonx still uses it. * Verified locally on main: the Bedrock integration module 99/99, with spotless. </details> ### API <!-- Does this change touches any public APIs? --> No new API; it uses `UnsupportedContentBlockException` from #1164. Compatibility: text-only requests are unchanged. User media, rejected since #1186, is now sent. ### Documentation <!-- Do not remove this section. Check the proper box only. --> - [ ] `doc-needed` <!-- Your PR changes impact docs --> - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [x] `doc-included` <!-- Your PR already contains the necessary documentation updates --> The Bedrock row of the Provider Capabilities table and a "Multimodal input" table under Amazon Bedrock in `providers.md`. ### Was this patch authored or co-authored using generative AI tooling? <!-- Do not remove this section. Check the proper box only. --> - [x] Yes - [ ] No If yes, include a `Generated-by: <tool name and version> (<model name and version>)` line, for example `Generated-by: Claude Code 2.1.226 (Claude Opus 4.6)`, in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html). Generated-by: Claude Code 2.1.259 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
