jbonofre opened a new pull request, #41506: URL: https://github.com/apache/beam/pull/41506
Similar to the recent JmsIO fixes (#40605, #41013), `SqsCheckpointMark` previously kept `private transient Optional<SqsUnboundedReader> reader`: 1. **Restored checkpoints failed with NPE**: After serialization/deserialization via `SerializableCoder`, the `transient` `reader` field becomes `null`, so `finalizeCheckpoint()` throws `NullPointerException` on `reader.isPresent()` and never deletes the messages from SQS or decrements `numInFlightCheckpoints`. 2. **Unfinalized earlier checkpoints & client leak**: Every `getCheckpointMark()` increments `numInFlightCheckpoints`, which is only decremented in `finalizeCheckpoint()`. Any skipped/unfinalized checkpoint (or checkpoint finalized on a restored copy) prevents `numInFlightCheckpoints` from reaching 0, leaking `sqsClient` on `close()`, and `SqsUnboundedSource.createReader(..., checkpointMark)` did not finalize the committed checkpoint on same-JVM resume. ## Changes - Add a per-reader weak-valued `PendingCheckpoints` registry (`readerId` + monotonic `sequence` on `SqsCheckpointMark`, with `serialVersionUID` pinned for backward compatibility). - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` (even from a restored copy), deletes `safeToDeleteIds` for this checkpoint and all earlier unfinalized checkpoints of the same reader, decrements `numInFlightCheckpoints` for all of them, and triggers `maybeCloseClient()`. If the originating reader is no longer in this JVM, it returns cleanly instead of throwing. - `SqsUnboundedSource.createReader(options, checkpointMark)` finalizes the resumed checkpoint when its originating reader lives in this JVM. - `SqsUnboundedReader.close()` schedules a 60s fallback timeout to abandon any unfinalized checkpoints and close `sqsClient` if finalization never arrives. - Added unit test `SqsUnboundedReaderTest.testRestoredCheckpointDeletesMessagesAndCatchesUpEarlierCheckpoints`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
