Efrat19 commented on code in PR #29954:
URL: https://github.com/apache/flink/pull/29954#discussion_r4580173004


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java:
##########
@@ -1894,7 +1894,18 @@ private void tryShutdownTimerService(long timeoutMs, 
TimerService timerService)
     public void dispatchOperatorEvent(OperatorID operator, 
SerializedValue<OperatorEvent> event)
             throws FlinkException {
         try {
+            // Operator events must be urgent: the coordinator considers an 
event delivered once it
+            // is enqueued here, and it may then take a checkpoint that must 
include the event's
+            // effect (e.g. AddSplitEvent). Unaligned checkpoint triggers are 
urgent mails (see
+            // triggerCheckpointAsync), so a normal mail could be overtaken by 
the trigger. Urgent
+            // mails are executed in FIFO order among themselves 
(TaskMailboxImpl keeps them in a
+            // dedicated FIFO urgent queue served before the batch; both mails 
are put with the
+            // same priority), which preserves the ordering.
+            // This is also safe for an aligned trigger (a normal mail): the 
coordinator's gateway
+            // stays closed from its checkpoint until the task's 
AcknowledgeCheckpointEvent, which
+            // is sent after the task-side snapshot, so later events can't 
overtake the snapshot.
             mainMailboxExecutor.execute(
+                    MailboxExecutor.MailOptions.urgent(),

Review Comment:
   Another bug?



##########
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskTest.java:
##########
@@ -1564,6 +1575,134 @@ void 
testTaskAvoidHangingAfterSnapshotStateThrownException() throws Exception {
         }
     }
 
+    /** Aligned-checkpoint timeout used by the {@code alignedWithTimeout} 
checkpoint options. */
+    private static final long ALIGNED_CHECKPOINT_TIMEOUT_MS = 10_000L;
+
+    /** Upper bound of mailbox steps to wait for the checkpoint trigger to 
complete. */
+    private static final int MAX_MAILBOX_STEPS = 100;
+
+    static Stream<Named<CheckpointOptions>> 
provideCheckpointOptionsForOperatorEventOrdering() {
+        return Stream.of(
+                Named.of(
+                        "alignedNoTimeout",
+                        CheckpointOptions.alignedNoTimeout(
+                                CheckpointType.CHECKPOINT, getDefault())),
+                Named.of(
+                        "alignedWithTimeout",
+                        CheckpointOptions.alignedWithTimeout(
+                                CheckpointType.CHECKPOINT,
+                                getDefault(),
+                                ALIGNED_CHECKPOINT_TIMEOUT_MS)),
+                Named.of(
+                        "unaligned",
+                        CheckpointOptions.unaligned(CheckpointType.CHECKPOINT, 
getDefault())));
+    }
+
+    /**
+     * Operator events dispatched before a checkpoint trigger must be applied 
before the operator is
+     * snapshotted, including the case of an urgent (unaligned) trigger. 
Events are urgent mails too
+     * so that they keep FIFO order with the urgent trigger.
+     */
+    @ParameterizedTest
+    @MethodSource("provideCheckpointOptionsForOperatorEventOrdering")
+    void testOperatorEventsAreAppliedBeforeCheckpointTrigger(CheckpointOptions 
checkpointOptions)

Review Comment:
   We seem to rely on operator events and barriers having the same numeric 
priority for correctness
   Could the test guard against future changes to that? e.g. tryYield() at 
operator priority after queueing and assert the snapshot hasn't run?



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -155,191 +179,134 @@ private Optional<Mail> tryTake(int priority, boolean 
ignoreDeferrable) {
         checkIsMailboxThread();
         checkTakeStateConditions();
 
-        moveUrgentMailsToBatchIfNeeded(true);
-
+        Mail urgentHead = takeUrgentIfPending(priority, false);
+        if (urgentHead != null) {
+            return urgentHead;
+        }
         Mail head = takeOrNull(batch, priority, false);
         if (head != null) {
             return head;
         }
         final ReentrantLock lock = this.lock;
         lock.lockInterruptibly();
         try {
-            Mail headMail;
-            while ((headMail = takeOrNull(queue, priority, false)) == null) {
+            while (true) {
+                Mail urgent = takeUrgentLocked(priority, false);
+                if (urgent != null) {
+                    return urgent;
+                }
+                Mail mail = takeNormalLocked(priority, false);
+                if (mail != null) {
+                    return mail;
+                }
                 // to ease debugging
                 notEmpty.await(1, TimeUnit.SECONDS);
             }
-            updateNewMailFlags();
-            return headMail;
         } finally {
             lock.unlock();
         }
     }
 
-    private void updateNewMailFlags() {
-        Mail peek = queue.peek();
-        if (peek != null) {
-            hasNewMail = true;
-            hasNewUrgentMail = peek.getMailOptions().isUrgent();
-        } else {
-            hasNewMail = false;
-            hasNewUrgentMail = false;
-        }
-    }
-
     // 
------------------------------------------------------------------------------------------------------------------
 
     @Override
     public boolean createBatch() {
-        moveUrgentMailsToBatchIfNeeded(false);
-        return !batch.isEmpty();
-    }
-
-    /**
-     * Try to create a batch of mails that can be taken with {@link 
#tryTakeFromBatch()}. The batch
-     * does not affect {@link #tryTake(int)} and {@link #take(int)}; that is, 
they return the same
-     * mails even if no batch had been created.
-     *
-     * <p>If a batch is not completely consumed by {@link 
#tryTakeFromBatch()}, its elements are
-     * carried over to the new batch.
-     *
-     * <p>Must be called from the mailbox thread ({@link #isMailboxThread()}.
-     *
-     * <p>To ensure that urgent mails are executed in FIFO order, moving mails 
from queue to batch
-     * only happens when batch doesn't have urgent mails. In addition, in 
order to reduce the
-     * frequent acquisition of locks caused by moving, moving is not required 
if batch is not empty
-     * and no new urgent mails.
-     *
-     * <pre>{@code
-     * All received mails (from head to tail):
-     *   Urgent mails:     MailA -> MailB -> MailC -> MailD -> MailE -> MailF
-     *   Non-urgent mails: Mail1 -> Mail2 -> Mail3 -> Mail4 -> Mail5 -> Mail6 
-> Mail7
-     *
-     * All received mails (The actual storage structure):
-     *   queue: UrgentMails and NonUrgentMails (UrgentMails is in reverse 
order)
-     *      MailF - MailE - MailD - Mail6 - Mail7 (the second batch)
-     *   batch: UrgentMails and NonUrgentMails (Both are FIFO)
-     *      MailA - MailB - MailC - Mail1 - Mail2 - Mail3 - Mail4 - Mail5 (the 
first batch)
-     *
-     * To ensure that urgent mails are executed in FIFO order, moving mails 
from queue to batch
-     * only happens when batch doesn't have urgent mails. For examples, the 
moving will happen when
-     * MailA, MailB and MailC are taken from batch:
-     *
-     * Step1 : moving all urgent mails into the head of batch:
-     *   queue: Mail6 - Mail7 - Mail8
-     *   batch: MailE - MailF - MailG - Mail1 - Mail2 - Mail3 - Mail4
-     * Step2 : moving all non-urgent mails into the tail of batch:
-     *   queue:
-     *   batch: MailE - MailF - MailG - Mail1 - Mail2 - Mail3 - Mail4 - Mail5 
- Mail6 - Mail7
-     * }</pre>
-     */
-    private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
         checkIsMailboxThread();
-        Mail peek = batch.peek();
-        if (peek != null) {
-            if (peek.getMailOptions().isUrgent()) {
-                // To ensure that urgent mails are executed in FIFO order, 
moving mails from queue
-                // to batch only happens when batch doesn't have urgent mails.
-                return;
-            } else if (!hasNewUrgentMail) {
-                // In order to reduce the frequent lock acquisition caused by 
movement, if the batch
-                // is not empty and there are no new urgent emails, there is 
no need to move.
-                return;
-            }
-        } else {
-            if (onlyMoveUrgentMails && !hasNewUrgentMail) {
-                return;
-            }
-            if (!hasNewMail) {
-                // Both batch and queue are empty
-                return;
-            }
-        }
-
-        final ReentrantLock lock = this.lock;
-        lock.lock();
-        try {
-            Mail mail;
-            while ((mail = queue.pollFirst()) != null) {
-                if (mail.getMailOptions().isUrgent()) {
-                    batch.addFirst(mail);
-                } else {
-                    if (onlyMoveUrgentMails) {
-                        // Put non-urgent mail back into the queue, and stop 
the loop
-                        queue.addFirst(mail);
-                        break;
-                    } else {
-                        batch.addLast(mail);
-                    }
-                }
+        if (hasNewMail) {
+            final ReentrantLock lock = this.lock;
+            lock.lock();
+            try {
+                batch.addAll(queue);
+                queue.clear();
+                hasNewMail = false;
+            } finally {
+                lock.unlock();
             }
-            hasNewUrgentMail = false;
-            hasNewMail = !queue.isEmpty();
-        } finally {
-            lock.unlock();
         }
+        // pending urgent mails are served by tryTakeFromBatch, too
+        return !batch.isEmpty() || hasNewUrgentMail;
     }
 
     @Override
     public Optional<Mail> tryTakeFromBatch() {
         checkIsMailboxThread();
         checkTakeStateConditions();
-        moveUrgentMailsToBatchIfNeeded(true);
+        Mail urgent = takeUrgentIfPending(Integer.MIN_VALUE, false);
+        if (urgent != null) {
+            return Optional.of(urgent);
+        }
         return Optional.ofNullable(batch.pollFirst());
     }
 
     // 
------------------------------------------------------------------------------------------------------------------
 
     @Override
     public void put(@Nonnull Mail mail) {
-        if (mail.getMailOptions().isUrgent()) {
-            putFirst(mail);
-        } else {
-            putLast(mail);
+        final boolean urgent = mail.getMailOptions().isUrgent();
+        final ReentrantLock lock = this.lock;
+        lock.lock();
+        try {
+            checkPutStateConditions();
+            if (urgent) {
+                final boolean wasEmpty = urgentQueue.isEmpty();
+                urgentQueue.addLast(mail);
+                if (wasEmpty) {
+                    hasNewUrgentMail = true;
+                }
+            } else {
+                final boolean wasEmpty = queue.isEmpty();
+                queue.addLast(mail);
+                if (wasEmpty) {
+                    hasNewMail = true;
+                }
+            }
+            notEmpty.signal();
+        } finally {
+            lock.unlock();
         }
     }
 
-    /** Adds the given action to the tail of the mailbox. */
-    private void putLast(@Nonnull Mail mail) {
+    // 
------------------------------------------------------------------------------------------------------------------
+
+    /**
+     * Takes the oldest eligible urgent mail, if any, without acquiring the 
lock when no urgent mail
+     * is pending according to the volatile {@link #hasNewUrgentMail}.
+     */
+    @Nullable
+    private Mail takeUrgentIfPending(int priority, boolean ignoreDeferrable) {
+        return hasNewUrgentMail ? takeUrgentOrNull(priority, ignoreDeferrable) 
: null;

Review Comment:
   wdyt about tracking the highest priority in the urgent queue instead of 
`hasNewUrgentMail`?
   It could save acquiring the lock when we do have urgent mails but they have 
a lower priority and would be skipped anyway
   ```suggestion
           return highestUrgentPriority >= priority ? 
takeUrgentOrNull(priority, ignoreDeferrable) : null;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to