haruki-830 opened a new pull request, #4520: URL: https://github.com/apache/flink-cdc/pull/4520
This PR adds multimodal AI function support to Flink CDC Pipeline transforms. `AI_IMAGE_COMPLETE` uses the standard OpenAI-compatible vision chat protocol. `AI_IMAGE_EMBED` and `SupportsImageEmbedding` are added as extension points, but image embedding is not yet supported by any built-in client. DashScope-specific embedding is intentionally out of scope. ## Brief change log - Added `SupportsImageTextGeneration` and `SupportsImageEmbedding`. - Added `AI_IMAGE_COMPLETE` and `AI_IMAGE_EMBED`. - Added runtime parsing, type inference, capability validation, and UDF precedence handling. - Implemented OpenAI-compatible vision chat using MIME detection and Base64 data URLs. - Added tests for supported image formats, Base64 encoding, null input, and invalid input. - Updated the English and Chinese documentation. ## Verifying this change This change added tests and can be verified as follows: - Updated `AiFunctionsTest` and `AiFunctionParserTest`. - Updated `OpenAiCompatibleModelClientTest` and `OpenAiCompatibleModelClientFactoryTest`. - Updated `FlinkPipelineAiFunctionITCase`. - Verified the tests with both the default Flink profile and `-Pflink2`. - Verified the complete repository with `mvn spotless:check`. ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs and JavaDocs --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (OpenAI Codex) Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
