ruanhang1993 commented on PR #3767: URL: https://github.com/apache/flink-cdc/pull/3767#issuecomment-2785528917
> > @JNSimba Could we provide this feature by `SupportsProjectionPushDown`? `debezium.column.include.list` and `debezium.column.exclude.list` is hard for users to understand and use. We could use `SupportsProjectionPushDown` interface to automatically generate the debezium setting. In this way we don't need to validate whether there is a conflict between the provided schema and the debezium setting. > > @ruanhang1993 `SupportsProjectionPushDown` is indeed an elegant approach, but some of the cdc logic seems to have problems in the push-down phase, for example, here > > If you run the `MySqlConnectorITCase.testConsumingAllEvents` case, the chunkkey will be adjusted here. For example, if my project column does not have a chunkkey, an error will be reported > > https://github.com/apache/flink-cdc/blob/b437a49e67a2ef84c59cde09f2bfc372f45b31fa/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java#L267C1-L274C37 > > ```java > Object[] chunkKey = > RecordUtils.getSplitKey( > splitKeyType, statefulTaskContext.getSchemaNameAdjuster(), target); > for (FinishedSnapshotSplitInfo splitInfo : finishedSplitsInfo.get(tableId)) { > if (RecordUtils.splitKeyRangeContains( > chunkKey, splitInfo.getSplitStart(), splitInfo.getSplitEnd()) > && position.isAfter(splitInfo.getHighWatermark())) { > return true; > ``` @JNSimba If the chunk key column is not contained in the projection, the snapshot phase will fail because of the lost of the chunk column. In this case, I think we need to contain these columns in snapshot phase and incremental phase. And apply the projection in the emitter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
