lvyanquan commented on code in PR #4418:
URL: https://github.com/apache/flink-cdc/pull/4418#discussion_r3570294026
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/assigners/MySqlHybridSplitAssigner.java:
##########
@@ -161,20 +179,75 @@ public void addSplits(Collection<MySqlSplit> splits) {
} else {
// we don't store the split, but will re-create binlog split
later
isBinlogSplitAssigned = false;
+ // re-creating the binlog split: the reader must re-assemble
and re-report
+ // before the snapshot metadata can be released again. Bumping
the generation
+ // invalidates any assembled event still in flight from the
failed attempt.
+ binlogSplitMetaAssembled = false;
+ checkpointIdToReleaseMeta = null;
+ binlogAssignmentGeneration++;
}
}
snapshotSplitAssigner.addSplits(snapshotSplits);
}
@Override
public PendingSplitsState snapshotState(long checkpointId) {
+ // Schedule releasing the snapshot metadata once the binlog split is
assigned and the reader
+ // has assembled it; the release happens in notifyCheckpointComplete.
Skipped when
+ // newly-added-table scan is on, since that flow may still need the
metadata.
+ if (isBinlogSplitAssigned
+ && binlogSplitMetaAssembled
+ && checkpointIdToReleaseMeta == null
+ && !snapshotSplitAssigner.isSnapshotMetaReleased()
+ && !sourceConfig.isScanNewlyAddedTableEnabled()) {
Review Comment:
This flag is not part of the checkpointed enumerator state. For a job that
has already started with `scan.newly-added-table.enabled = false`, once this PR
releases the snapshot metadata, enabling `scan.newly-added-table.enabled` later
for the same job becomes incompatible: the old assignedSplits,
splitFinishedOffsets, and tableSchemas are already gone, while the
newly-added-table flow may still need them to update the binlog split.
This changes the previous behavior, where the metadata was always kept in
state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
