jbonofre opened a new pull request, #41509:
URL: https://github.com/apache/beam/pull/41509
Similar to the recent JmsIO fixes (#40605, #41013), `PubsubCheckpoint`
previously lost its `reader` and `safeToAckIds` when encoded and decoded
through `PubsubCheckpointCoder`:
1. **Restored checkpoints failed with IllegalStateException**:
`PubsubCheckpointCoder.decode` sets `reader = null` and `safeToAckIds = null`,
so calling `finalizeCheckpoint()` on a restored mark throws
`IllegalStateException("Cannot finalize a restored checkpoint")`, even when the
originating `PubsubReader` is still live in the same JVM.
2. **Unfinalized checkpoints leaked `numInFlightCheckpoints` & resume did
not finalize**: Every `getCheckpointMark()` increments
`numInFlightCheckpoints`, which is only decremented in `finalizeCheckpoint()`.
Any skipped/unfinalized checkpoint keeps `numInFlightCheckpoints > 0` so
`maybeCloseClient()` in `close()` never closes `pubsubClient`, and
`PubsubSource.createReader(..., checkpoint)` did not finalize the resumed
checkpoint on same-JVM resume.
## Changes
- Add a per-reader weak-valued `PendingCheckpoints` registry (`readerId` +
monotonic `sequence` encoded by `PubsubCheckpointCoder`).
- `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints`
(even from a restored copy), acknowledges `safeToAckIds` for this checkpoint
and all earlier unfinalized checkpoints of the same reader, decrements
`numInFlightCheckpoints` for all of them, and triggers `maybeCloseClient()`. If
the originating reader is no longer in this JVM, it returns cleanly instead of
throwing.
- `PubsubSource.createReader(options, checkpoint)` finalizes the resumed
checkpoint when its originating reader lives in this JVM before nacking
`notYetReadIds`.
- `PubsubReader.close()` schedules a 60s fallback timeout to abandon any
unfinalized checkpoints and close `pubsubClient` if finalization never arrives.
- Added unit test
`PubsubUnboundedSourceTest.restoredCheckpointFinalizesLiveMessagesAndCatchesUpEarlierCheckpoints`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]