Yanquan Lv created FLINK-36913:
----------------------------------
Summary: Add an option in Kafka Sink to manual map table to topic
name.
Key: FLINK-36913
URL: https://issues.apache.org/jira/browse/FLINK-36913
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Affects Versions: cdc-3.3.0
Reporter: Yanquan Lv
Fix For: cdc-3.3.0
For many users who used Kafka as DataSink in YAML job, them usually want to
send changelog to one topic, in current framework, we can use `route` module to
do that, like what the following YAML file does:
{noformat}
source:
type: mysql
hostname: xxxxxx
port: 3306
username: flink
password: Flinkxxx
tables: flink_source.news_[0-9]
sink:
type: kafka
properties.bootstrap.servers: xxx:9092
value.format: canal-json
route:
- source-table: flink_source.news_[0-9]
sink-table: my_source
{noformat}
However, the out put of Kafka in canal-json format doesn't contain the original
database/table information, instead, it only contains the database/table
information after routing. Although this is in line with the functionality of
route, it does not meet the needs of users.
Therefore, I suggest adding a parameter in the sink to let Kafka determine how
to handle the mapping from table to topic name, so we can create one topic for
many source tables, and keep all database/table information of source tables
when output records.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
