[
https://issues.apache.org/jira/browse/FLINK-40513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mao Jiayi reopened FLINK-40513:
-------------------------------
> Introduce multimodal AI functions for image understanding
> ---------------------------------------------------------
>
> Key: FLINK-40513
> URL: https://issues.apache.org/jira/browse/FLINK-40513
> Project: Flink
> Issue Type: New Feature
> Components: Flink CDC
> Reporter: Mao Jiayi
> Priority: Major
> Labels: pull-request-available
>
> This issue adds multimodal AI functions to the transform framework, so users
> can process image columns directly in transform expressions.
> Two {{@Experimental}} capability interfaces are introduced in
> {{{}flink-cdc-common{}}}:
> * {{SupportsImageTextGeneration}} — generates text from image bytes and a
> prompt.
> * {{SupportsImageEmbedding}} — converts image bytes to a float vector.
> Two built-in functions are registered:
> * {{AI_IMAGE_COMPLETE(model, image, prompt)}} returns {{{}STRING{}}}. The
> OpenAI-compatible provider implements this via standard vision chat, encoding
> images as Base64 data URLs.
> * {{AI_IMAGE_EMBED(model, image)}} returns {{{}ARRAY<FLOAT>{}}}. This
> function is provided as a framework extension point only. The default
> OpenAI-compatible provider does not implement image embedding, and the
> community distribution does not yet include a production-ready provider.
> Users who need image embedding must wait for a follow-up PR.
> Image MIME type is detected from magic bytes (PNG, JPEG, GIF, WebP). A NULL
> image returns NULL without calling the provider. Empty byte arrays and
> unrecognized formats are rejected before the HTTP request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
