jbonofre opened a new pull request, #41507:
URL: https://github.com/apache/beam/pull/41507

   Similar to the recent JmsIO fixes (#40605, #41013), `AmqpCheckpointMark` 
previously kept `transient Messenger messenger` and `transient List<Tracker> 
trackers`, and `UnboundedAmqpReader` reused a single mutable 
`AmqpCheckpointMark` across all `getCheckpointMark()` calls:
   
   1. **Restored checkpoints lost pending trackers**: After 
serialization/deserialization via `SerializableCoder`, `readObject` resets 
`trackers` to an empty list and leaves `messenger` `null`, so 
`finalizeCheckpoint()` on a restored mark silently acknowledges nothing.
   2. **No per-checkpoint isolation, catch-up, or deferred close**: Checkpoints 
taken without being finalized (e.g., dropped callbacks or progress snapshots) 
are not caught up on subsequent finalization or on `createReader(..., 
checkpointMark)` resume, and `UnboundedAmqpReader.close()` immediately stops 
`messenger` before in-flight checkpoints finalize.
   
   ## Changes
   - Isolate per-checkpoint tracker snapshots via `AmqpCheckpointMark.Preparer` 
and register them in a per-reader weak-valued `PendingCheckpoints` registry 
(`readerId` + monotonic `sequence` on `AmqpCheckpointMark`, with 
`serialVersionUID` pinned for backward compatibility).
   - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` 
(even from a restored copy) and acknowledges this checkpoint together with all 
earlier unfinalized checkpoints of the same reader in order.
   - `UnboundedAmqpSource.createReader(options, checkpointMark)` finalizes the 
checkpoint it resumes from when its originating reader lives in this JVM.
   - `UnboundedAmqpReader.close()` waits for in-flight checkpoints to finalize 
(up to a 60s timeout, closing immediately once drained) before stopping 
`messenger`.
   - Added unit test 
`AmqpIOTest.testRestoredCheckpointAcknowledgesLiveTrackersAndEarlierCheckpoints`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to