jbonofre opened a new pull request, #41509:
URL: https://github.com/apache/beam/pull/41509

   Similar to the recent JmsIO fixes (#40605, #41013), `PubsubCheckpoint` 
previously lost its `reader` and `safeToAckIds` when encoded and decoded 
through `PubsubCheckpointCoder`:
   
   1. **Restored checkpoints failed with IllegalStateException**: 
`PubsubCheckpointCoder.decode` sets `reader = null` and `safeToAckIds = null`, 
so calling `finalizeCheckpoint()` on a restored mark throws 
`IllegalStateException("Cannot finalize a restored checkpoint")`, even when the 
originating `PubsubReader` is still live in the same JVM.
   2. **Unfinalized checkpoints leaked `numInFlightCheckpoints` & resume did 
not finalize**: Every `getCheckpointMark()` increments 
`numInFlightCheckpoints`, which is only decremented in `finalizeCheckpoint()`. 
Any skipped/unfinalized checkpoint keeps `numInFlightCheckpoints > 0` so 
`maybeCloseClient()` in `close()` never closes `pubsubClient`, and 
`PubsubSource.createReader(..., checkpoint)` did not finalize the resumed 
checkpoint on same-JVM resume.
   
   ## Changes
   - Add a per-reader weak-valued `PendingCheckpoints` registry (`readerId` + 
monotonic `sequence` encoded by `PubsubCheckpointCoder`).
   - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` 
(even from a restored copy), acknowledges `safeToAckIds` for this checkpoint 
and all earlier unfinalized checkpoints of the same reader, decrements 
`numInFlightCheckpoints` for all of them, and triggers `maybeCloseClient()`. If 
the originating reader is no longer in this JVM, it returns cleanly instead of 
throwing.
   - `PubsubSource.createReader(options, checkpoint)` finalizes the resumed 
checkpoint when its originating reader lives in this JVM before nacking 
`notYetReadIds`.
   - `PubsubReader.close()` schedules a 60s fallback timeout to abandon any 
unfinalized checkpoints and close `pubsubClient` if finalization never arrives.
   - Added unit test 
`PubsubUnboundedSourceTest.restoredCheckpointFinalizesLiveMessagesAndCatchesUpEarlierCheckpoints`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] [PubsubIO] Acknowledge li... jbonofre (via GitHub) via github

Reply via email to