jbonofre opened a new pull request, #41506:
URL: https://github.com/apache/beam/pull/41506

   Similar to the recent JmsIO fixes (#40605, #41013), `SqsCheckpointMark` 
previously kept `private transient Optional<SqsUnboundedReader> reader`:
   
   1. **Restored checkpoints failed with NPE**: After 
serialization/deserialization via `SerializableCoder`, the `transient` `reader` 
field becomes `null`, so `finalizeCheckpoint()` throws `NullPointerException` 
on `reader.isPresent()` and never deletes the messages from SQS or decrements 
`numInFlightCheckpoints`.
   2. **Unfinalized earlier checkpoints & client leak**: Every 
`getCheckpointMark()` increments `numInFlightCheckpoints`, which is only 
decremented in `finalizeCheckpoint()`. Any skipped/unfinalized checkpoint (or 
checkpoint finalized on a restored copy) prevents `numInFlightCheckpoints` from 
reaching 0, leaking `sqsClient` on `close()`, and 
`SqsUnboundedSource.createReader(..., checkpointMark)` did not finalize the 
committed checkpoint on same-JVM resume.
   
   ## Changes
   - Add a per-reader weak-valued `PendingCheckpoints` registry (`readerId` + 
monotonic `sequence` on `SqsCheckpointMark`, with `serialVersionUID` pinned for 
backward compatibility).
   - `finalizeCheckpoint()` resolves the live reader's `PendingCheckpoints` 
(even from a restored copy), deletes `safeToDeleteIds` for this checkpoint and 
all earlier unfinalized checkpoints of the same reader, decrements 
`numInFlightCheckpoints` for all of them, and triggers `maybeCloseClient()`. If 
the originating reader is no longer in this JVM, it returns cleanly instead of 
throwing.
   - `SqsUnboundedSource.createReader(options, checkpointMark)` finalizes the 
resumed checkpoint when its originating reader lives in this JVM.
   - `SqsUnboundedReader.close()` schedules a 60s fallback timeout to abandon 
any unfinalized checkpoints and close `sqsClient` if finalization never arrives.
   - Added unit test 
`SqsUnboundedReaderTest.testRestoredCheckpointDeletesMessagesAndCatchesUpEarlierCheckpoints`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to