[
https://issues.apache.org/jira/browse/FLINK-36813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Di Wu updated FLINK-36813:
--------------------------
Description:
*Background*
In some scenarios, MySQL synchronization only expects to synchronize specified
fields instead of all fields in the table.
1. The user only has the permission for some fields in MySQL
2. The user has too many fields in a single table and only wants to synchronize
some fields, for example, here
[https://github.com/apache/flink-cdc/discussions/3058]
*Current situation*
For the incremental stage, you only need to configure the column.include.list
property of debezium to support the synchronization of some fields in the
incremental stage, refer to:
[https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-property-column-include-list]
For the full snapshot stage, * is currently used in
{_}MySqlSnapshotSplitReadTask{_}, refer to
{code:java}
if (isScanningData) {
return buildSelectWithRowLimits(
tableId, limitSize, "*", Optional.ofNullable(condition), Optional.empty());
{code}
*Solution*
We can refer to debezium
[RelationalSnapshotChangeEventSource](https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/relational/RelationalSnapshotChangeEventSource.java#L752-L776),
The user configures column.include.list, and then captures the specific
columns in MySqlSnapshotSplitReadTask, and splices them when constructing Scan
SQL.
was:
*Background*
In some scenarios, MySQL synchronization only expects to synchronize specified
fields instead of all fields in the table.
1. The user only has the permission for some fields in MySQL
2. The user has too many fields in a single table and only wants to synchronize
some fields, for example, here
https://github.com/apache/flink-cdc/discussions/3058
*Current situation*
For the incremental stage, you only need to configure the column.include.list
property of debezium to support the synchronization of some fields in the
incremental stage, refer to:
https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-property-column-include-list
For the full snapshot stage, * is currently used in
{_}MySqlSnapshotSplitReadTask{_}, refer to
{code:java}
if (isScanningData) {
return buildSelectWithRowLimits(
tableId, limitSize, "*", Optional.ofNullable(condition), Optional.empty());
{code}
*Solution*
The user configures {_}column.include.list{_}, and then captures the specific
columns in MySqlSnapshotSplitReadTask, and splices them when constructing Scan
SQL.
> MySQLCDC supports synchronization of specified fields
> -----------------------------------------------------
>
> Key: FLINK-36813
> URL: https://issues.apache.org/jira/browse/FLINK-36813
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.1
> Reporter: Di Wu
> Priority: Major
> Labels: CDC, pull-request-available
> Fix For: cdc-3.3.0
>
>
> *Background*
> In some scenarios, MySQL synchronization only expects to synchronize
> specified fields instead of all fields in the table.
> 1. The user only has the permission for some fields in MySQL
> 2. The user has too many fields in a single table and only wants to
> synchronize some fields, for example, here
> [https://github.com/apache/flink-cdc/discussions/3058]
> *Current situation*
> For the incremental stage, you only need to configure the column.include.list
> property of debezium to support the synchronization of some fields in the
> incremental stage, refer to:
> [https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-property-column-include-list]
> For the full snapshot stage, * is currently used in
> {_}MySqlSnapshotSplitReadTask{_}, refer to
> {code:java}
> if (isScanningData) {
> return buildSelectWithRowLimits(
> tableId, limitSize, "*", Optional.ofNullable(condition), Optional.empty());
> {code}
>
> *Solution*
> We can refer to debezium
> [RelationalSnapshotChangeEventSource](https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/relational/RelationalSnapshotChangeEventSource.java#L752-L776),
> The user configures column.include.list, and then captures the specific
> columns in MySqlSnapshotSplitReadTask, and splices them when constructing
> Scan SQL.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
