GitHub user Zhuoxi2000 added a comment to the discussion: [Discuss][API]
Multimodal content parts in ChatMessage
Thanks for the detailed review. Agreed on both points. I also think replacing
`content: String` directly is cleaner than keeping two representations. Since
the API is still Beta, this feels like the right time to make that change.
## 1. Public data model
I’d go with modality specific blocks as the public API:
`TextBlock`, `ImageBlock`, `AudioBlock`, `VideoBlock`, and `DocumentBlock`.
Media blocks would share the same basic fields: `mime_type`, either `data` or
`url`, plus optional metadata such as `name`, `size_bytes`, and `sha256`.
`ChatMessage` would then use an ordered `List<ContentBlock>` /
`List[ContentBlock]` as the source of truth.
Existing helpers like `user(String)` would keep working by creating a single
`TextBlock`, and `getText()` / `.text` would provide the concatenated text
projection for existing text consumers.
Java and Python would use the same serialized shape with a `type` discriminator:
```json
{
"role": "user",
"blocks": [
{"type": "text", "text": "What's in this picture?"},
{"type": "image", "mime_type": "image/png", "data": "<base64>"}
]
}
```
A few choices I’d make explicit:
* Media uses base64 `data` or `url`, exactly one, with `mime_type` required.
* Block order is preserved through serialization and provider conversion.
* TOOL messages use the same block model. This also gives us a clean path to
fix MCP image results, which are currently stringified before reaching the
model.
* No dual representation. `content` goes away and `blocks` becomes the source
of truth.
The cross language snapshots, pemja bridge, and existing call sites would all
be migrated together.
## 2. Provider integration
The flow would stay simple:
`ChatMessage.blocks -> provider converter -> provider native content`
Each `ChatModelConnection` handles the conversion in its existing message
conversion path.
For example, OpenAI maps image blocks to `image_url`, Anthropic maps them to
image source blocks, and Ollama maps them to its `images` field.
If a provider does not support a block type, I’d fail fast with a clear
validation error instead of silently dropping or converting it. A capability
API can come later if we want preflight checks.
The same model also works for responses and tool results, so we do not need
another representation later when providers start returning multimodal content.
Java and Python would follow the same block model and wire contract, with
mirrored converter tests for providers supported in both runtimes.
GitHub link:
https://github.com/apache/flink-agents/discussions/1031#discussioncomment-18178994
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
