uicosp commented on code in PR #3776:
URL: https://github.com/apache/flink-cdc/pull/3776#discussion_r4323976201
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/task/MySqlSnapshotSplitReadTask.java:
##########
@@ -242,12 +247,21 @@ private void createDataEventsForTable(
long exportStart = clock.currentTimeInMillis();
LOG.info("Exporting data from split '{}' of table {}",
snapshotSplit.splitId(), table.id());
+ if (snapshotFilter != null) {
+ LOG.info(
+ "Filter for split '{}' of table {} is: {}",
+ snapshotSplit.splitId(),
+ table.id(),
+ snapshotFilter);
+ }
+
final String selectSql =
StatementUtils.buildSplitScanQuery(
snapshotSplit.getTableId(),
snapshotSplit.getSplitKeyType(),
snapshotSplit.getSplitStart() == null,
- snapshotSplit.getSplitEnd() == null);
+ snapshotSplit.getSplitEnd() == null,
+ snapshotFilter);
Review Comment:
Agreed, the intended semantics needed clarification. The filter is intended
to apply only to the snapshot SELECT query. Binlog events, including those
backfilled between the low and high watermarks, remain unfiltered, so the final
normalized snapshot is not guaranteed to satisfy the predicate. Updated the
documentation and API descriptions to clarify this behavior.
Added concurrent DML tests covering rows entering and leaving the filter
condition before the high watermark, with snapshot backfill both enabled and
skipped. The tests pass on Flink 1.20 and 2.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
