github-actions[bot] commented on code in PR #41506:
URL: https://github.com/apache/beam/pull/41506#discussion_r4645015807


##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsCheckpointMark.java:
##########
@@ -68,32 +88,150 @@ class SqsCheckpointMark implements 
UnboundedSource.CheckpointMark, Serializable
       SqsUnboundedReader reader,
       Collection<String> messagesToDelete,
       Collection<String> notYetReadReceipts) {
+    this(null, 0, reader, messagesToDelete, notYetReadReceipts);
+  }
+
+  SqsCheckpointMark(
+      @Nullable PendingCheckpoints pendingCheckpoints,
+      long sequence,
+      SqsUnboundedReader reader,
+      Collection<String> messagesToDelete,
+      Collection<String> notYetReadReceipts) {
+    this.pendingCheckpoints = pendingCheckpoints;
+    this.readerId = pendingCheckpoints == null ? null : 
pendingCheckpoints.readerId;
+    this.sequence = sequence;
     this.reader = Optional.of(reader);
     this.safeToDeleteIds = ImmutableList.copyOf(messagesToDelete);
     this.notYetReadReceipts = ImmutableList.copyOf(notYetReadReceipts);
   }
 
   @Override
   public void finalizeCheckpoint() throws IOException {
-    checkState(
-        reader.isPresent() && safeToDeleteIds != null, "Cannot finalize a 
restored checkpoint");
+    PendingCheckpoints pending = pendingCheckpoints;
+    if (pending == null && readerId != null) {
+      pending = PendingCheckpoints.LIVE.getIfPresent(readerId);
+    }
+    if (pending != null) {
+      List<SqsCheckpointMark> checkpoints = pending.removeUpTo(sequence);
+      if (checkpoints.isEmpty()) {
+        return;
+      }
+      Set<String> toDelete = new LinkedHashSet<>();
+      for (SqsCheckpointMark checkpoint : checkpoints) {
+        if (checkpoint.safeToDeleteIds != null) {
+          toDelete.addAll(checkpoint.safeToDeleteIds);
+        }
+      }
+      try {
+        pending.reader.delete(new ArrayList<>(toDelete));

Review Comment:
   **🟠 High**: `pending.reader.delete(...)` is called outside `closeLock` and 
before `numInFlightCheckpoints` is decremented, while the close timeout task in 
`SqsUnboundedReader.close()` (or a concurrent `finalizeCheckpoint()` after 
timeout) can concurrently decrement `numInFlightCheckpoints` to `<= 0` and call 
`maybeCloseClient()`, setting `sqsClient = null` and closing the client. If a 
late `finalizeCheckpoint()` races with the close timeout:
   1. `finalizeCheckpoint()` calls `pending.removeUpTo(sequence)` and gets the 
checkpoints.
   2. The close timeout task fires, calls `pendingCheckpoints.removeAll()` (now 
empty), and calls `maybeCloseClient()`.
   3. If another checkpoint finishes or if the timeout task closes `sqsClient`, 
`pending.reader.delete(...)` will throw a `NullPointerException` when 
dereferencing `sqsClient` in `deleteBatch` (or use a closed client).
   
   Even simpler: if checkpoint 1 is removed by `removeUpTo(1)` right before the 
timeout task runs, and checkpoint 2 was still in `pendingCheckpoints`, the 
timeout task removes checkpoint 2, decrements `numInFlightCheckpoints` by 1, 
and then when checkpoint 1 finishes `delete()` and enters `finally`, 
`clearAndDecrement()` decrements `numInFlightCheckpoints` to 0 and 
`maybeCloseClient()` closes `sqsClient`. Conversely, if only checkpoint 1 
existed, `removeUpTo(1)` empties `pendingCheckpoints`, then the timeout task 
runs `removeAll()` (returns 0 items) and calls `maybeCloseClient()` (does not 
close yet because `numInFlightCheckpoints` is still 1), OR if `abandon` in the 
timeout task clears checkpoints without coordinating with in-progress 
`delete()` calls, `clearAndDecrement()` on an abandoned checkpoint that was 
concurrently removed would double-decrement `numInFlightCheckpoints` if not 
careful. Moreover, in `SqsUnboundedReader.close()`, if the timeout fires while 
no `removeUpTo` is ru
 nning, it calls `pendingCheckpoints.removeAll()`, decrements 
`numInFlightCheckpoints` to 0, and closes `sqsClient`, but does NOT clear 
`safeToDeleteIds` / `reader` on the abandoned `SqsCheckpointMark` instances! If 
a direct reference to one of those `SqsCheckpointMark` instances has 
`pendingCheckpoints == null` (or if `clearAndDecrement` is later called), or if 
`removeAll()` raced with `finalizeCheckpoint()`, `sqsClient` may already be 
`null`.
   
   Consider clearing/abandoning each checkpoint mark inside the timeout task 
(via a synchronized method on `SqsCheckpointMark` similar to 
`JmsCheckpointMark.abandon()`) and guarding `delete()` / client closure so 
`delete()` safely skips or aborts if the reader client is already closed.



##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsUnboundedReader.java:
##########
@@ -520,21 +539,57 @@ public boolean advance() throws IOException {
    * no in-flight checkpoints remain, the reader will be closed.
    */
   @Override
+  @SuppressWarnings("FutureReturnValueIgnored")
   public void close() throws IOException {
     active.set(false);
-    maybeCloseClient();
+    synchronized (closeLock) {
+      if (numInFlightCheckpoints.get() == 0) {
+        maybeCloseClient();
+      } else if (closeTimeoutFuture == null) {
+        ScheduledExecutorService executorService =
+            awsOptions.as(ExecutorOptions.class).getScheduledExecutorService();
+        closeTimeoutFuture =
+            executorService.schedule(
+                () -> {
+                  List<SqsCheckpointMark> abandoned = 
pendingCheckpoints.removeAll();

Review Comment:
   **🟠 High**: When the close timeout expires and 
`pendingCheckpoints.removeAll()` is called, the abandoned `SqsCheckpointMark` 
instances themselves are not marked as cleared/abandoned (`safeToDeleteIds` and 
`reader` remain non-null on them). Although `pendingCheckpoints` is non-null on 
the in-memory mark (so a subsequent `finalizeCheckpoint()` on it will see 
`pending.removeUpTo(sequence)` return empty), `clearAndDecrement()` is never 
called on the abandoned marks, leaving their `reader` 
(`Optional<SqsUnboundedReader>`) and `safeToDeleteIds` references retained in 
memory if the runner holds onto the `CheckpointMark` instance, and risking a 
negative `numInFlightCheckpoints` (`IllegalStateException: Miscounted in-flight 
checkpoints`) if `clearAndDecrement()` is ever invoked on them. Calling 
`checkpoint.clearAndDecrement()` for each checkpoint in `abandoned` (instead of 
manually doing `numInFlightCheckpoints.addAndGet(-abandoned.size())`) will 
atomically decrement the counter once pe
 r checkpoint and release references.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to