github-actions[bot] commented on code in PR #41508:
URL: https://github.com/apache/beam/pull/41508#discussion_r4645144770
##########
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java:
##########
@@ -410,71 +427,256 @@ public boolean requiresDeduping() {
}
}
- private static class RabbitMQCheckpointMark
- implements UnboundedSource.CheckpointMark, Serializable {
- transient Channel channel;
+ @VisibleForTesting
+ static class RabbitMQCheckpointMark implements
UnboundedSource.CheckpointMark, Serializable {
+
+ // Pinned to the value computed before readerId and sequence were added,
so checkpoints written
+ // by older versions can still be decoded.
+ private static final long serialVersionUID = -961062677995876719L;
+
+ private final @Nullable String readerId;
+ private final long sequence;
+ transient @Nullable Channel channel;
Instant latestTimestamp = Instant.now();
transient ConcurrentLinkedQueue<Long> sessionIds = new
ConcurrentLinkedQueue<>();
+ transient @Nullable PendingCheckpoints pendingCheckpoints;
+ transient @Nullable AtomicInteger activeCheckpoints;
+ transient @Nullable Runnable onCheckpointFinalized;
+
+ RabbitMQCheckpointMark() {
+ this(null, 0, null, Instant.now(), new ConcurrentLinkedQueue<>(), null,
null);
+ }
+
+ private RabbitMQCheckpointMark(
+ @Nullable PendingCheckpoints pendingCheckpoints,
+ long sequence,
+ @Nullable Channel channel,
+ Instant latestTimestamp,
+ ConcurrentLinkedQueue<Long> sessionIds,
+ @Nullable AtomicInteger activeCheckpoints,
+ @Nullable Runnable onCheckpointFinalized) {
+ this.pendingCheckpoints = pendingCheckpoints;
+ this.readerId = pendingCheckpoints == null ? null :
pendingCheckpoints.readerId;
+ this.sequence = sequence;
+ this.channel = channel;
+ this.latestTimestamp = latestTimestamp;
+ this.sessionIds = sessionIds;
+ this.activeCheckpoints = activeCheckpoints;
+ this.onCheckpointFinalized = onCheckpointFinalized;
+ }
// this method is called after deserialization on the deserialized object
- private Object readResolve() {
+ private synchronized Object readResolve() {
// (re-)initialize transient fields as required
this.sessionIds = new ConcurrentLinkedQueue<>();
return this;
}
- /**
- * Advances the watermark to the provided time, provided said time is
after the current
- * watermark. If the provided time is before the latest, this function
no-ops.
- *
- * @param time The time to advance the watermark to
- */
- public void advanceWatermark(Instant time) {
- if (time.isAfter(latestTimestamp)) {
- latestTimestamp = time;
- }
- }
-
@Override
public void finalizeCheckpoint() throws IOException {
- Long sessionId = sessionIds.poll();
- while (sessionId != null) {
- channel.basicAck(sessionId, false);
- sessionId = sessionIds.poll();
+ PendingCheckpoints pending = pendingCheckpoints;
+ if (pending == null && readerId != null) {
+ pending = PendingCheckpoints.LIVE.getIfPresent(readerId);
+ }
+ if (pending != null) {
+ for (RabbitMQCheckpointMark checkpoint : pending.removeUpTo(sequence))
{
+ checkpoint.acknowledge();
+ }
+ return;
}
- if (channel.isOpen()) {
- channel.txCommit();
+ if (readerId != null) {
+ return;
+ }
+ acknowledge();
+ }
+
+ private synchronized void acknowledge() throws IOException {
+ Channel ch = channel;
Review Comment:
**🟠High**: In `acknowledge()` and `finalizeCheckpoint()`:
1. `ch.basicAck(sessionId, false)` is called without checking `ch.isOpen()`,
and neither `basicAck` nor `txCommit` is wrapped in a `try ... finally` that
clears `sessionIds`/`channel` and calls `release(true)`. If `basicAck` or
`txCommit` throws `IOException` or `AlreadyClosedException`,
`activeCheckpoints` is never decremented.
2. In `finalizeCheckpoint()`, `pending.removeUpTo(sequence)` removes all
checkpoints up to `sequence` before iterating. If an earlier checkpoint's
`acknowledge()` throws, the remaining checkpoints in the removed list are never
acknowledged and never have `release(true)` called, permanently leaking their
`activeCheckpoints` counts and preventing `maybeStopConnection()` from closing
the connection until the 60s timeout.
3. Because `pending.removeUpTo(sequence)` removes the checkpoint before
`acknowledge()` runs, a concurrent timeout firing
`abandonPendingCheckpointsAndStopConnection()` will not see the in-flight
checkpoint in `pendingCheckpoints.removeAll()` and will call
`connectionHandler.stop()` concurrently with `ch.basicAck`/`ch.txCommit`.
##########
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java:
##########
@@ -395,8 +406,14 @@ public List<RabbitMQSource> split(int desiredNumSplits,
PipelineOptions options)
@Override
public UnboundedReader<RabbitMqMessage> createReader(
- PipelineOptions options, RabbitMQCheckpointMark checkpointMark) throws
IOException {
- return new UnboundedRabbitMqReader(this, checkpointMark);
+ PipelineOptions options, @Nullable RabbitMQCheckpointMark
checkpointMark)
+ throws IOException {
+ if (checkpointMark != null) {
+ // A runner resumes from a checkpoint only once it is committed, so
its messages can be
+ // acknowledged if the reader that took it lives in this JVM.
+ checkpointMark.finalizeCheckpoint();
+ }
Review Comment:
**🟠High**: When a runner calls `createReader(options, checkpointMark)` to
resume from a committed checkpoint, it is often recovering because the previous
reader's connection or channel failed. Because the previous reader's `close()`
defers `abandonPendingCheckpointsAndStopConnection()` for 60s while
`activeCheckpoints > 0`, `checkpointMark` will still resolve the old reader in
`PendingCheckpoints.LIVE` and attempt `basicAck`/`txCommit` on the dead
channel. If that throws `IOException` or `AlreadyClosedException` (a
`RuntimeException`), `createReader` will fail to create the new replacement
reader instead of logging and letting the broker redeliver any unacknowledged
messages (as `JmsCheckpointMark` does).
##########
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java:
##########
@@ -410,71 +427,256 @@ public boolean requiresDeduping() {
}
}
- private static class RabbitMQCheckpointMark
- implements UnboundedSource.CheckpointMark, Serializable {
- transient Channel channel;
+ @VisibleForTesting
+ static class RabbitMQCheckpointMark implements
UnboundedSource.CheckpointMark, Serializable {
+
+ // Pinned to the value computed before readerId and sequence were added,
so checkpoints written
+ // by older versions can still be decoded.
+ private static final long serialVersionUID = -961062677995876719L;
+
+ private final @Nullable String readerId;
+ private final long sequence;
+ transient @Nullable Channel channel;
Instant latestTimestamp = Instant.now();
transient ConcurrentLinkedQueue<Long> sessionIds = new
ConcurrentLinkedQueue<>();
+ transient @Nullable PendingCheckpoints pendingCheckpoints;
+ transient @Nullable AtomicInteger activeCheckpoints;
+ transient @Nullable Runnable onCheckpointFinalized;
+
+ RabbitMQCheckpointMark() {
+ this(null, 0, null, Instant.now(), new ConcurrentLinkedQueue<>(), null,
null);
+ }
+
+ private RabbitMQCheckpointMark(
+ @Nullable PendingCheckpoints pendingCheckpoints,
+ long sequence,
+ @Nullable Channel channel,
+ Instant latestTimestamp,
+ ConcurrentLinkedQueue<Long> sessionIds,
+ @Nullable AtomicInteger activeCheckpoints,
+ @Nullable Runnable onCheckpointFinalized) {
+ this.pendingCheckpoints = pendingCheckpoints;
+ this.readerId = pendingCheckpoints == null ? null :
pendingCheckpoints.readerId;
+ this.sequence = sequence;
+ this.channel = channel;
+ this.latestTimestamp = latestTimestamp;
+ this.sessionIds = sessionIds;
+ this.activeCheckpoints = activeCheckpoints;
+ this.onCheckpointFinalized = onCheckpointFinalized;
+ }
// this method is called after deserialization on the deserialized object
- private Object readResolve() {
+ private synchronized Object readResolve() {
// (re-)initialize transient fields as required
this.sessionIds = new ConcurrentLinkedQueue<>();
return this;
}
- /**
- * Advances the watermark to the provided time, provided said time is
after the current
- * watermark. If the provided time is before the latest, this function
no-ops.
- *
- * @param time The time to advance the watermark to
- */
- public void advanceWatermark(Instant time) {
- if (time.isAfter(latestTimestamp)) {
- latestTimestamp = time;
- }
- }
-
@Override
public void finalizeCheckpoint() throws IOException {
- Long sessionId = sessionIds.poll();
- while (sessionId != null) {
- channel.basicAck(sessionId, false);
- sessionId = sessionIds.poll();
+ PendingCheckpoints pending = pendingCheckpoints;
+ if (pending == null && readerId != null) {
+ pending = PendingCheckpoints.LIVE.getIfPresent(readerId);
+ }
+ if (pending != null) {
+ for (RabbitMQCheckpointMark checkpoint : pending.removeUpTo(sequence))
{
+ checkpoint.acknowledge();
+ }
+ return;
}
- if (channel.isOpen()) {
- channel.txCommit();
+ if (readerId != null) {
+ return;
+ }
+ acknowledge();
+ }
+
+ private synchronized void acknowledge() throws IOException {
+ Channel ch = channel;
+ if (ch != null) {
+ Long sessionId = sessionIds.poll();
+ while (sessionId != null) {
+ ch.basicAck(sessionId, false);
+ sessionId = sessionIds.poll();
+ }
+ if (ch.isOpen()) {
+ ch.txCommit();
+ }
+ } else {
+ sessionIds.clear();
}
+ channel = null;
latestTimestamp = Instant.now();
+ release(true);
Review Comment:
**🟡 Medium**: `latestTimestamp` is a non-transient field serialized by
`SerializableCoder`. `Read.UnboundedSourceAsSDFWrapperFn` caches active readers
using `restrictionCoder.structuralValue(...)` (which encodes the
`RabbitMQCheckpointMark` to bytes). Mutating `latestTimestamp = Instant.now()`
when `acknowledge()` runs changes the serialized bytes of an existing
`RabbitMQCheckpointMark` instance in memory, which can cause a cache miss in
`cachedReaders` on the next bundle if the same checkpoint instance is held in
the restriction. Since `getWatermark()` now reads
`checkpointPreparer.getLatestTimestamp()`, `mark.latestTimestamp` should not be
mutated after checkpoint creation.
##########
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java:
##########
@@ -410,71 +427,256 @@ public boolean requiresDeduping() {
}
}
- private static class RabbitMQCheckpointMark
- implements UnboundedSource.CheckpointMark, Serializable {
- transient Channel channel;
+ @VisibleForTesting
+ static class RabbitMQCheckpointMark implements
UnboundedSource.CheckpointMark, Serializable {
+
+ // Pinned to the value computed before readerId and sequence were added,
so checkpoints written
+ // by older versions can still be decoded.
+ private static final long serialVersionUID = -961062677995876719L;
+
+ private final @Nullable String readerId;
+ private final long sequence;
+ transient @Nullable Channel channel;
Instant latestTimestamp = Instant.now();
transient ConcurrentLinkedQueue<Long> sessionIds = new
ConcurrentLinkedQueue<>();
+ transient @Nullable PendingCheckpoints pendingCheckpoints;
+ transient @Nullable AtomicInteger activeCheckpoints;
+ transient @Nullable Runnable onCheckpointFinalized;
+
+ RabbitMQCheckpointMark() {
+ this(null, 0, null, Instant.now(), new ConcurrentLinkedQueue<>(), null,
null);
+ }
+
+ private RabbitMQCheckpointMark(
+ @Nullable PendingCheckpoints pendingCheckpoints,
+ long sequence,
+ @Nullable Channel channel,
+ Instant latestTimestamp,
+ ConcurrentLinkedQueue<Long> sessionIds,
+ @Nullable AtomicInteger activeCheckpoints,
+ @Nullable Runnable onCheckpointFinalized) {
+ this.pendingCheckpoints = pendingCheckpoints;
+ this.readerId = pendingCheckpoints == null ? null :
pendingCheckpoints.readerId;
+ this.sequence = sequence;
+ this.channel = channel;
+ this.latestTimestamp = latestTimestamp;
+ this.sessionIds = sessionIds;
+ this.activeCheckpoints = activeCheckpoints;
+ this.onCheckpointFinalized = onCheckpointFinalized;
+ }
// this method is called after deserialization on the deserialized object
- private Object readResolve() {
+ private synchronized Object readResolve() {
// (re-)initialize transient fields as required
this.sessionIds = new ConcurrentLinkedQueue<>();
return this;
}
- /**
- * Advances the watermark to the provided time, provided said time is
after the current
- * watermark. If the provided time is before the latest, this function
no-ops.
- *
- * @param time The time to advance the watermark to
- */
- public void advanceWatermark(Instant time) {
- if (time.isAfter(latestTimestamp)) {
- latestTimestamp = time;
- }
- }
-
@Override
public void finalizeCheckpoint() throws IOException {
- Long sessionId = sessionIds.poll();
- while (sessionId != null) {
- channel.basicAck(sessionId, false);
- sessionId = sessionIds.poll();
+ PendingCheckpoints pending = pendingCheckpoints;
+ if (pending == null && readerId != null) {
+ pending = PendingCheckpoints.LIVE.getIfPresent(readerId);
+ }
+ if (pending != null) {
+ for (RabbitMQCheckpointMark checkpoint : pending.removeUpTo(sequence))
{
+ checkpoint.acknowledge();
+ }
+ return;
}
- if (channel.isOpen()) {
- channel.txCommit();
+ if (readerId != null) {
+ return;
+ }
+ acknowledge();
+ }
+
+ private synchronized void acknowledge() throws IOException {
+ Channel ch = channel;
+ if (ch != null) {
+ Long sessionId = sessionIds.poll();
+ while (sessionId != null) {
+ ch.basicAck(sessionId, false);
+ sessionId = sessionIds.poll();
+ }
+ if (ch.isOpen()) {
+ ch.txCommit();
+ }
+ } else {
+ sessionIds.clear();
}
+ channel = null;
latestTimestamp = Instant.now();
+ release(true);
+ }
+
+ private synchronized void abandon() {
+ sessionIds.clear();
+ channel = null;
+ release(false);
+ }
+
+ private void release(boolean notifyIfDrained) {
+ if (activeCheckpoints != null) {
+ int remaining = activeCheckpoints.decrementAndGet();
+ activeCheckpoints = null;
+ if (notifyIfDrained && remaining == 0 && onCheckpointFinalized !=
null) {
+ onCheckpointFinalized.run();
+ }
+ }
+ }
+
+ private static final class PendingCheckpoints {
+ private static final Cache<String, PendingCheckpoints> LIVE =
+ CacheBuilder.newBuilder().weakValues().build();
+
+ private final String readerId = UUID.randomUUID().toString();
+ private final List<RabbitMQCheckpointMark> checkpoints = new
ArrayList<>();
+ private long nextSequence;
+
+ static PendingCheckpoints create() {
+ PendingCheckpoints pending = new PendingCheckpoints();
+ LIVE.put(pending.readerId, pending);
+ return pending;
+ }
+
+ synchronized long nextSequence() {
+ return nextSequence++;
+ }
+
+ synchronized void add(RabbitMQCheckpointMark checkpoint) {
+ checkpoints.add(checkpoint);
+ }
+
+ synchronized List<RabbitMQCheckpointMark> removeUpTo(long sequence) {
+ List<RabbitMQCheckpointMark> removed = new ArrayList<>();
+ checkpoints.removeIf(
+ checkpoint -> {
+ if (checkpoint.sequence <= sequence) {
+ removed.add(checkpoint);
+ return true;
+ }
+ return false;
+ });
+ return removed;
+ }
+
+ synchronized List<RabbitMQCheckpointMark> removeAll() {
+ LIVE.invalidate(readerId);
+ List<RabbitMQCheckpointMark> removed = new ArrayList<>(checkpoints);
+ checkpoints.clear();
+ return removed;
+ }
+ }
+
+ static class Preparer {
+ private final PendingCheckpoints pendingCheckpoints =
PendingCheckpoints.create();
+ private final List<Long> sessionIds = new ArrayList<>();
+ private volatile Instant latestTimestamp = Instant.now();
+ private boolean discarded = false;
+
+ synchronized void add(long deliveryTag) {
+ if (discarded) {
+ throw new IllegalStateException("Attempting to add delivery tag to
discarded checkpoint");
+ }
+ sessionIds.add(deliveryTag);
+ }
+
+ /**
+ * Advances the watermark to the provided time, provided said time is
after the current
+ * watermark. If the provided time is before the latest, this function
no-ops.
+ */
+ synchronized void advanceWatermark(Instant time) {
+ if (time.isAfter(latestTimestamp)) {
+ latestTimestamp = time;
+ }
+ }
+
+ Instant getLatestTimestamp() {
+ return latestTimestamp;
+ }
+
+ synchronized void discard() {
+ sessionIds.clear();
+ discarded = true;
+ }
+
+ synchronized RabbitMQCheckpointMark newCheckpoint(
+ @Nullable Channel channel,
+ @Nullable AtomicInteger activeCheckpoints,
+ @Nullable Runnable onCheckpointFinalized) {
+ if (discarded || sessionIds.isEmpty()) {
+ sessionIds.clear();
+ return new RabbitMQCheckpointMark(
+ pendingCheckpoints,
+ pendingCheckpoints.nextSequence(),
+ null,
+ latestTimestamp,
+ new ConcurrentLinkedQueue<>(),
+ null,
+ null);
+ }
+ ConcurrentLinkedQueue<Long> snapshot = new
ConcurrentLinkedQueue<>(sessionIds);
+ sessionIds.clear();
+ Instant checkpointTimestamp = latestTimestamp;
+ latestTimestamp = Instant.now();
+ RabbitMQCheckpointMark mark =
Review Comment:
**🟢 Low**: Unconditionally resetting `latestTimestamp = Instant.now()` in
`newCheckpoint()` can cause `getWatermark()` to move backwards if a message had
a timestamp slightly ahead of `Instant.now()` (e.g., due to producer clock
skew). Using `advanceWatermark(Instant.now())` preserves monotonicity.
```suggestion
advanceWatermark(Instant.now());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]