github-actions[bot] commented on code in PR #41506:
URL: https://github.com/apache/beam/pull/41506#discussion_r4645015807
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsCheckpointMark.java:
##########
@@ -68,32 +88,150 @@ class SqsCheckpointMark implements
UnboundedSource.CheckpointMark, Serializable
SqsUnboundedReader reader,
Collection<String> messagesToDelete,
Collection<String> notYetReadReceipts) {
+ this(null, 0, reader, messagesToDelete, notYetReadReceipts);
+ }
+
+ SqsCheckpointMark(
+ @Nullable PendingCheckpoints pendingCheckpoints,
+ long sequence,
+ SqsUnboundedReader reader,
+ Collection<String> messagesToDelete,
+ Collection<String> notYetReadReceipts) {
+ this.pendingCheckpoints = pendingCheckpoints;
+ this.readerId = pendingCheckpoints == null ? null :
pendingCheckpoints.readerId;
+ this.sequence = sequence;
this.reader = Optional.of(reader);
this.safeToDeleteIds = ImmutableList.copyOf(messagesToDelete);
this.notYetReadReceipts = ImmutableList.copyOf(notYetReadReceipts);
}
@Override
public void finalizeCheckpoint() throws IOException {
- checkState(
- reader.isPresent() && safeToDeleteIds != null, "Cannot finalize a
restored checkpoint");
+ PendingCheckpoints pending = pendingCheckpoints;
+ if (pending == null && readerId != null) {
+ pending = PendingCheckpoints.LIVE.getIfPresent(readerId);
+ }
+ if (pending != null) {
+ List<SqsCheckpointMark> checkpoints = pending.removeUpTo(sequence);
+ if (checkpoints.isEmpty()) {
+ return;
+ }
+ Set<String> toDelete = new LinkedHashSet<>();
+ for (SqsCheckpointMark checkpoint : checkpoints) {
+ if (checkpoint.safeToDeleteIds != null) {
+ toDelete.addAll(checkpoint.safeToDeleteIds);
+ }
+ }
+ try {
+ pending.reader.delete(new ArrayList<>(toDelete));
Review Comment:
**🟠High**: `pending.reader.delete(...)` is called outside `closeLock` and
before `numInFlightCheckpoints` is decremented, while the close timeout task in
`SqsUnboundedReader.close()` (or a concurrent `finalizeCheckpoint()` after
timeout) can concurrently decrement `numInFlightCheckpoints` to `<= 0` and call
`maybeCloseClient()`, setting `sqsClient = null` and closing the client. If a
late `finalizeCheckpoint()` races with the close timeout:
1. `finalizeCheckpoint()` calls `pending.removeUpTo(sequence)` and gets the
checkpoints.
2. The close timeout task fires, calls `pendingCheckpoints.removeAll()` (now
empty), and calls `maybeCloseClient()`.
3. If another checkpoint finishes or if the timeout task closes `sqsClient`,
`pending.reader.delete(...)` will throw a `NullPointerException` when
dereferencing `sqsClient` in `deleteBatch` (or use a closed client).
Even simpler: if checkpoint 1 is removed by `removeUpTo(1)` right before the
timeout task runs, and checkpoint 2 was still in `pendingCheckpoints`, the
timeout task removes checkpoint 2, decrements `numInFlightCheckpoints` by 1,
and then when checkpoint 1 finishes `delete()` and enters `finally`,
`clearAndDecrement()` decrements `numInFlightCheckpoints` to 0 and
`maybeCloseClient()` closes `sqsClient`. Conversely, if only checkpoint 1
existed, `removeUpTo(1)` empties `pendingCheckpoints`, then the timeout task
runs `removeAll()` (returns 0 items) and calls `maybeCloseClient()` (does not
close yet because `numInFlightCheckpoints` is still 1), OR if `abandon` in the
timeout task clears checkpoints without coordinating with in-progress
`delete()` calls, `clearAndDecrement()` on an abandoned checkpoint that was
concurrently removed would double-decrement `numInFlightCheckpoints` if not
careful. Moreover, in `SqsUnboundedReader.close()`, if the timeout fires while
no `removeUpTo` is ru
nning, it calls `pendingCheckpoints.removeAll()`, decrements
`numInFlightCheckpoints` to 0, and closes `sqsClient`, but does NOT clear
`safeToDeleteIds` / `reader` on the abandoned `SqsCheckpointMark` instances! If
a direct reference to one of those `SqsCheckpointMark` instances has
`pendingCheckpoints == null` (or if `clearAndDecrement` is later called), or if
`removeAll()` raced with `finalizeCheckpoint()`, `sqsClient` may already be
`null`.
Consider clearing/abandoning each checkpoint mark inside the timeout task
(via a synchronized method on `SqsCheckpointMark` similar to
`JmsCheckpointMark.abandon()`) and guarding `delete()` / client closure so
`delete()` safely skips or aborts if the reader client is already closed.
##########
sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sqs/SqsUnboundedReader.java:
##########
@@ -520,21 +539,57 @@ public boolean advance() throws IOException {
* no in-flight checkpoints remain, the reader will be closed.
*/
@Override
+ @SuppressWarnings("FutureReturnValueIgnored")
public void close() throws IOException {
active.set(false);
- maybeCloseClient();
+ synchronized (closeLock) {
+ if (numInFlightCheckpoints.get() == 0) {
+ maybeCloseClient();
+ } else if (closeTimeoutFuture == null) {
+ ScheduledExecutorService executorService =
+ awsOptions.as(ExecutorOptions.class).getScheduledExecutorService();
+ closeTimeoutFuture =
+ executorService.schedule(
+ () -> {
+ List<SqsCheckpointMark> abandoned =
pendingCheckpoints.removeAll();
Review Comment:
**🟠High**: When the close timeout expires and
`pendingCheckpoints.removeAll()` is called, the abandoned `SqsCheckpointMark`
instances themselves are not marked as cleared/abandoned (`safeToDeleteIds` and
`reader` remain non-null on them). Although `pendingCheckpoints` is non-null on
the in-memory mark (so a subsequent `finalizeCheckpoint()` on it will see
`pending.removeUpTo(sequence)` return empty), `clearAndDecrement()` is never
called on the abandoned marks, leaving their `reader`
(`Optional<SqsUnboundedReader>`) and `safeToDeleteIds` references retained in
memory if the runner holds onto the `CheckpointMark` instance, and risking a
negative `numInFlightCheckpoints` (`IllegalStateException: Miscounted in-flight
checkpoints`) if `clearAndDecrement()` is ever invoked on them. Calling
`checkpoint.clearAndDecrement()` for each checkpoint in `abandoned` (instead of
manually doing `numInFlightCheckpoints.addAndGet(-abandoned.size())`) will
atomically decrement the counter once pe
r checkpoint and release references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]