jbonofre opened a new pull request, #41508:
URL: https://github.com/apache/beam/pull/41508

   Similar to the recent JmsIO fixes (#40605, #41013), `RabbitMQCheckpointMark` 
previously kept `transient Channel channel` and `transient 
ConcurrentLinkedQueue<Long> sessionIds`, and `UnboundedRabbitMqReader` reused a 
single mutable `RabbitMQCheckpointMark` across all `getCheckpointMark()` calls:
   
   1. **Restored checkpoints lost delivery tags and threw NPE**: After 
serialization/deserialization via `SerializableCoder`, `readResolve` resets 
`sessionIds` to empty and leaves `channel` `null`, so `finalizeCheckpoint()` on 
a restored mark loses all pending acks and throws `NullPointerException` on 
`channel.isOpen()`.
   2. **No per-checkpoint isolation, catch-up, or deferred close**: Checkpoints 
taken without being finalized (e.g., dropped callbacks or progress snapshots) 
are not caught up on subsequent finalization or on `createReader(..., 
checkpointMark)` resume, and `UnboundedRabbitMqReader.close()` immediately 
stops `connectionHandler` before in-flight checkpoints finalize.
   
   ## Changes
   - Isolate per-checkpoint delivery tags via `RabbitMQCheckpointMark.Preparer` 
and register them in a per-reader weak-valued `PendingCheckpoints` registry 
(`readerId` + monotonic `sequence` on `RabbitMQCheckpointMark`, with 
`serialVersionUID` pinned for backward compatibility).
   - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` 
(even from a restored copy) and acknowledges/commits this checkpoint together 
with all earlier unfinalized checkpoints of the same reader in order.
   - `RabbitMQSource.createReader(options, checkpointMark)` finalizes the 
checkpoint it resumes from when its originating reader lives in this JVM.
   - `UnboundedRabbitMqReader.close()` waits for in-flight checkpoints to 
finalize (up to a 60s timeout, closing immediately once drained) before 
stopping `connectionHandler`.
   - Added unit test 
`RabbitMqIOTest.testRestoredCheckpointAcknowledgesLiveAndEarlierCheckpoints`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to