Zhuoxi2000 opened a new pull request, #1275: URL: https://github.com/apache/flink-agents/pull/1275
<!-- * Thank you very much for contributing to Flink Agents. * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc. --> <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. --> Linked issue: #1059 ### Purpose of change <!-- What is the purpose of this change? --> A user `ChatMessage` with image or document blocks now reaches the OpenAI Responses API with its media, replacing the rejection #1186 added. This is the #1059 "OpenAI Responses API" follow-up, after #1164 (Chat Completions), #1178 (Ollama) and #1204 (Anthropic). The Responses connection is Java-only, so there is no Python part. #### Runtime flow 1. `OpenAIResponsesUtils.convertSingleMessage` maps a user message's blocks in order to Responses input content: text, images and documents; any other block throws. 2. Media in an assistant message or a tool result throws. System messages cannot hold media since #1185. 3. A user message without media is sent exactly as before, as one `input_text` with its text projection. #### Key decisions * Same shape as #1164: images by URL or as a `data:` URI; Base64 documents as `file_data` with the block's `name` as `filename` (`document` when unset), paired as in #1164 and OpenAI's file-input examples. * Documents by URL use `input_file.file_url`, which the Responses API has and Chat Completions lacks; `filename` is sent only when the block has a name. * `input_image.detail` is required by openai-java 4.8.0 and is sent as `auto`, the API default. * Image and file types are not restricted locally; OpenAI rejects what a model does not accept, as in #1164. * Audio and video are rejected: the 4.8.0 `ResponseInputContent` union has only text, image and file. * Tool results stay text-only (`function_call_output.output` is a string in 4.8.0). * `chat()` has no wrapping `try`, so `UnsupportedContentBlockException` reaches callers with its own type. ### Behavioral Semantics <!-- For a non-trivial code change whose implementation is largely AI-assisted: interaction decisions, behavioral contracts, and failure behavior. See `contribution-guides/ai-assisted-pr.md`. Remove this heading and this comment otherwise. --> #### Interaction decisions | Role | Blocks | Result | |---|---|---| | user | text only | unchanged: one `input_text` | | user | images and documents, with or without text | input content in block order | | user | audio or video | `UnsupportedContentBlockException`; nothing sent | | assistant, or a tool result | any media | `UnsupportedContentBlockException`; nothing sent | #### Behavioral contracts 1. A user message without media is sent exactly as before. 2. A user message with media is sent as input content in block order, one `input_text` per `TextBlock`. 3. `ImageBlock` becomes `input_image` with `image_url` set to the URL or `data:<media_type>;base64,<data>`, and `detail: auto`. 4. `DocumentBlock` becomes `input_file`: Base64 data as `file_data` (a `data:` URI) with `filename` = the block's `name`, else `document`; a URL as `file_url`, with `filename` only when named. 5. `AudioBlock` and `VideoBlock`, and media in an assistant message or a tool result, throw `UnsupportedContentBlockException` naming the block but not its data or URL. 6. That exception reaches callers of `chat()` with its own type. #### Failure behavior * Unsupported media or role: thrown while building the request, so nothing is sent; the error is deterministic, so RETRY fails the same way. * A model or request OpenAI rejects (a non-vision model, an unsupported file type, an unreachable URL): the provider's error propagates as before. ### Tests <!-- How is this change verified? --> | Contract | `OpenAIResponsesMultimodalTest` | |---|---| | 1 | `testTextOnlyUserMessageIsUnchanged` | | 2, 3 | `testImagesBecomeInputImages` | | 4 | `testDocumentsBecomeInputFiles` | | 5 | `testUnsupportedBlocksFailExplicitly`, `testMediaOutsideUserMessagesFails` | | 6 | `testChatKeepsMediaErrorTypes` | Tests assert on the SDK-serialized input items. #1186's `testMediaBlocksFailExplicitly` in `OpenAIResponsesModelConnectionTest` is removed; `testRequestBuildingFailurePropagatesUnwrapped` still passes, now with the unsupported-block error. Not verified: a live OpenAI endpoint, which needs an API key. <details> <summary>Implementation invariants and supporting evidence</summary> * openai-java 4.8.0: `ResponseInputImage.Builder` has `imageUrl`, `fileId` and a required `detail`; `ResponseInputFile.Builder` has `fileData`, `fileUrl`, `fileId` and `filename`; `ResponseInputContent` is `input_text` | `input_image` | `input_file`. * `UnsupportedContentBlockException.rejectMedia` stays: Gemini, Bedrock and watsonx still use it. * Verified locally on main: the Java OpenAI integration module 153/153, with spotless. </details> ### API <!-- Does this change touches any public APIs? --> No new API; it uses `UnsupportedContentBlockException` from #1164. Compatibility: text-only requests are unchanged. User images and documents, rejected since #1186, are now sent. ### Documentation <!-- Do not remove this section. Check the proper box only. --> - [ ] `doc-needed` <!-- Your PR changes impact docs --> - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [x] `doc-included` <!-- Your PR already contains the necessary documentation updates --> A block table under Responses API in `providers.md`; the Responses API removed from the not-yet-supported lists in `chat.md` and `providers.md`. ### Was this patch authored or co-authored using generative AI tooling? <!-- Do not remove this section. Check the proper box only. --> - [x] Yes - [ ] No If yes, include a `Generated-by: <tool name and version> (<model name and version>)` line, for example `Generated-by: Claude Code 2.1.226 (Claude Opus 4.6)`, in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html). Generated-by: Claude Code 2.1.259 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
