zhangshenghang opened a new pull request, #12163:
URL: https://github.com/apache/seatunnel/pull/12163
## Purpose
`IncrementalSourceReader.reportFinishedSnapshotSplitsIfNeed` used to clear
`finishedUnackedSplits` immediately after sending
`CompletedSnapshotSplitsReportEvent` to the enumerator, with a TODO
acknowledging that the enumerator should send an ack. The enumerator never sent
one, so the reader lost track of which split watermarks had actually been
durably received.
A reader that reports all its completed snapshot splits and is then
preempted (static-slot failover) before the enumerator has processed the report
loses the information entirely: the new owner has no record of which splits
were finished, and the snapshot phase is stuck waiting for a report from a
reader that no longer exists.
## Changes
- `IncrementalSourceReader`:
- Drop the eager `finishedUnackedSplits.clear()` after the report is sent.
The splits now stay in `finishedUnackedSplits` until the enumerator returns a
`CompletedSnapshotSplitsAckEvent`.
- Override `handleSourceEvent` to consume
`CompletedSnapshotSplitsAckEvent` and remove the ack'd split IDs from
`finishedUnackedSplits`. All other source events are forwarded to the super
class.
- Add the corresponding `SourceEvent` and
`CompletedSnapshotSplitsAckEvent` imports.
The reader no longer loses ack'd state on failover, and the enumerator
becomes the single source of truth for which snapshot splits are completed.
## Validation
```
./mvnw -pl seatunnel-connectors-v2/connector-cdc/connector-cdc-base compile \
-Dcheckstyle.skip -Dspotless.check.skip
```
Result: BUILD SUCCESS. The existing `SnapshotSplitAssignerTest` continues to
pass on its own branch.
This is the third of three sub-PRs split out from the original ST-3491 fix.
The previous two are:
- #12162: reconstruct the missing snapshot completion watermark on restore,
and always wait for a durable checkpoint before marking the assigner completed.
- (this PR): stop dropping the finished-splits map on the reader side until
the enumerator has ack'd.
## Impact
- Behavior change: `finishedUnackedSplits` now grows for as long as the
enumerator has not yet ack'd, and shrinks only when the ack event arrives. On a
healthy run this is invisible (the ack comes back promptly). On a failover the
new owner of the static slot inherits the still-unacked splits and the report
is re-driven on the next poll.
- No new public API. `CompletedSnapshotSplitsAckEvent` is already in the
codebase; this PR just wires it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
