Zhuoxi2000 opened a new pull request, #1060: URL: https://github.com/apache/flink-agents/pull/1060
<!-- * Thank you very much for contributing to Flink Agents. * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc. --> <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. --> Linked issue: #1059 (Phase 1 — the tracking issue stays open for the provider phase) ### Purpose of change <!-- What is the purpose of this change? --> Implements the framework part of the multimodal design agreed in #1031. `ChatMessage.content: String` is replaced by `blocks: List<ContentBlock>` in both Java and Python. The initial block types are `TextBlock`, `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`. Media blocks share a MIME-typed shape with `mime_type`, exactly one of base64 `data` or external `url`, and optional metadata such as `name`, `size_bytes`, and `sha256`. Existing text constructors/factories keep their signatures and create a single `TextBlock`. `getText()` / `.text` returns the ordered text projection. Java and Python use the same `type`-discriminated JSON format, covered by the regenerated cross-language snapshots. The pemja bridge is updated to carry blocks across runtimes as well. This PR also updates prompt handling so image-only messages are preserved, and `Prompt.formatMessages` applies substitutions only to text blocks while passing media blocks through unchanged. Provider-specific multimodal conversion is intentionally left to the next #1059 phase. Existing providers continue using the text projection in this PR. Behavior changes: * CEL expressions using `response.content` need to migrate to `response.blocks`. * Python `ChatMessage` now rejects unknown fields, so the removed `content=` argument fails explicitly instead of silently creating an empty message. * `toString()` / `__str__`, equality, and hashing now use the block representation. * `data` and `url` are mutually exclusive for media blocks. ### Tests <!-- How is this change verified? --> * Added Java and Python `ChatMessage` serialization tests covering the wire format, mixed block ordering, media fields, round trips, source validation, and rejection of the removed `content` argument. * Regenerated the Java/Python cross-language snapshots using the existing snapshot flows; cross-deserialization passes in both directions. * Added durable-state serde coverage for a mixed text + image `ChatMessage`. * Java reactor tests and Python tests pass locally, aside from environment-specific dependency failures that are covered by CI. ### API Yes. This is a Beta breaking change discussed in #1031. `ChatMessage.content` is replaced by `blocks` in both Java and Python. Java removes `getContent()/setContent`; Python removes `.content`. Text-only access remains available through `getText()` / `.text` and `setText()` / `set_text()`. New public types in both languages are `ContentBlock`, `TextBlock`, `MediaBlock`, `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`. `ChatMessage` also adds `getBlocksAsMaps()` / `setBlocksFromMaps()` for the pemja bridge, where blocks cross as plain maps. CEL expressions referencing `response.content` must migrate to `response.blocks`. ### Documentation <!-- Do not remove this section. Check the proper box only. --> - [ ] `doc-needed` <!-- Your PR changes impact docs --> - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [x] `doc-included` <!-- Your PR already contains the necessary documentation updates --> ### Was this patch authored or co-authored using generative AI tooling? <!-- Do not remove this section. Check the proper box only. --> - [x] Yes - [ ] No Generated by: Claude Code Fable 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
