jbonofre opened a new pull request, #41508: URL: https://github.com/apache/beam/pull/41508
Similar to the recent JmsIO fixes (#40605, #41013), `RabbitMQCheckpointMark` previously kept `transient Channel channel` and `transient ConcurrentLinkedQueue<Long> sessionIds`, and `UnboundedRabbitMqReader` reused a single mutable `RabbitMQCheckpointMark` across all `getCheckpointMark()` calls: 1. **Restored checkpoints lost delivery tags and threw NPE**: After serialization/deserialization via `SerializableCoder`, `readResolve` resets `sessionIds` to empty and leaves `channel` `null`, so `finalizeCheckpoint()` on a restored mark loses all pending acks and throws `NullPointerException` on `channel.isOpen()`. 2. **No per-checkpoint isolation, catch-up, or deferred close**: Checkpoints taken without being finalized (e.g., dropped callbacks or progress snapshots) are not caught up on subsequent finalization or on `createReader(..., checkpointMark)` resume, and `UnboundedRabbitMqReader.close()` immediately stops `connectionHandler` before in-flight checkpoints finalize. ## Changes - Isolate per-checkpoint delivery tags via `RabbitMQCheckpointMark.Preparer` and register them in a per-reader weak-valued `PendingCheckpoints` registry (`readerId` + monotonic `sequence` on `RabbitMQCheckpointMark`, with `serialVersionUID` pinned for backward compatibility). - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` (even from a restored copy) and acknowledges/commits this checkpoint together with all earlier unfinalized checkpoints of the same reader in order. - `RabbitMQSource.createReader(options, checkpointMark)` finalizes the checkpoint it resumes from when its originating reader lives in this JVM. - `UnboundedRabbitMqReader.close()` waits for in-flight checkpoints to finalize (up to a 60s timeout, closing immediately once drained) before stopping `connectionHandler`. - Added unit test `RabbitMqIOTest.testRestoredCheckpointAcknowledgesLiveAndEarlierCheckpoints`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
