loserwang1024 commented on code in PR #4608:
URL: https://github.com/apache/flink-cdc/pull/4608#discussion_r4477471581
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/main/java/org/apache/flink/cdc/connectors/fluss/source/event/TableSubscriptionEvent.java:
##########
@@ -35,21 +36,20 @@ public class TableSubscriptionEvent implements SourceEvent {
private final Set<TablePath> subscribedTablePaths;
private final Map<TablePath, Long> pendingRemovalRequests;
private final Set<TablePath> fencedTablePaths;
-
- public TableSubscriptionEvent(
- Set<TablePath> subscribedTablePaths, Map<TablePath, Long>
pendingRemovalRequests) {
- this(subscribedTablePaths, pendingRemovalRequests,
Collections.emptySet());
- }
+ private final Set<PhysicalTablePath> subscribedPhysicalTablePaths;
Review Comment:
This PR addresses recovery after partition removal, but splits for
partitions deleted during normal execution can still remain active and be
included in subsequent checkpoints.
Could we extend the existing table-removal request/ACK mechanism to
partition-level removals in a follow-up, using a common cleanup lifecycle for
both recovery and runtime discovery? This should cover unsubscribing affected
buckets, cancelling pending or ongoing snapshot reads, and finishing the
corresponding splits before acknowledging removal, together with cleanup of the
enumerator’s assignment state.
This would also give us an opportunity to revisit whether the full
physical-partition inventory is still needed for recovery. I’m fine with
keeping that refactoring outside the scope of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
