Gopal Dirisala created FLINK-40325:
--------------------------------------
Summary: Support Kafka as a CDC Source Connector in Flink CDC
Pipeline
Key: FLINK-40325
URL: https://issues.apache.org/jira/browse/FLINK-40325
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.5.0, cdc-3.6.0, cdc-3.7.0
Reporter: Gopal Dirisala
h2. Summary
Add support for *Kafka as a source connector* in the Flink CDC Pipeline,
enabling CDC events published to Kafka (e.g., Debezium JSON, Canal JSON) to be
consumed and processed using the existing Pipeline API.
h2. Motivation
The Flink CDC Pipeline currently supports:
* Database → Kafka
* Database → Iceberg
However, it does not support:
{{Database
│
Flink CDC
│
Kafka
│
Flink CDC Pipeline
│
Iceberg}}
Organizations commonly use Kafka as a durable CDC event bus to decouple
producers and consumers, enable replay, and support multiple downstream
applications.
Today, users must leave the Flink CDC Pipeline framework and implement a
separate Flink SQL or DataStream application to consume Kafka CDC events and
write them to downstream systems.
h2. Proposed Enhancement
Introduce a *Kafka Source Connector* for the Flink CDC Pipeline that supports
CDC formats such as:
* Debezium JSON
* Canal JSON
The connector should convert Kafka CDC events into the internal Flink CDC
changelog representation, allowing them to be processed by existing pipeline
transformations and sink connectors.
h2. Benefits
* Enables Kafka as a first-class CDC source in the Pipeline framework.
* Preserves a unified Pipeline programming model for end-to-end CDC workflows.
* Eliminates the need for separate Flink SQL/DataStream applications for Kafka
→ Iceberg (or other sinks).
* Supports common architectures where Kafka acts as the central CDC transport
layer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
