Izeren commented on code in PR #29954:
URL: https://github.com/apache/flink/pull/29954#discussion_r4612202215


##########
flink-core/src/main/java/org/apache/flink/api/common/operators/MailboxExecutor.java:
##########
@@ -89,21 +89,35 @@ public interface MailboxExecutor {
     /** Extra options to configure enqueued mails. */
     @PublicEvolving
     interface MailOptions {
-        static MailOptions options() {
-            return MailOptionsImpl.DEFAULT;
-        }
 
         /**
-         * Mark this mail as deferrable.
-         *
-         * <p>Runtime can decide to defer execution of deferrable mails. For 
example, to unblock

Review Comment:
   !nit you have changed formatting here. 



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/MailboxProcessor.java:
##########
@@ -330,15 +330,17 @@ private void sendPoisonMail(RunnableWithException mail) {
     }
 
     /**
-     * Sends the given <code>mail</code> using {@link 
TaskMailbox#putFirst(Mail)} . Intended use is
-     * to control this <code>MailboxProcessor</code>; no interaction with 
tasks should be performed;
+     * Sends the given <code>mail</code> with urgent. Intended use is to 
control this <code>
+     * MailboxProcessor</code>; no interaction with tasks should be performed. 
The mail is served
+     * before all non-urgent mails, but after urgent mails that are already 
pending (FIFO).
      */
     private void sendControlMail(
             RunnableWithException mail, String descriptionFormat, Object... 
descriptionArgs) {
-        mailbox.putFirst(
+        mailbox.put(

Review Comment:
   Are we no longer treating control mails as top priority? Or is it top 
priority except for urgent? 
   
   As I understand, the original code (before Rui's changes) could actually 
have control messages overtaking each other in case 2 threads were racing to 
put control mail before mailbox thread was waken up. Do we change the logic to 
strict FIFO here and place new message after any other that also has 
MAX_PRIORITY?



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
 /**
  * Implementation of {@link TaskMailbox} in a {@link 
java.util.concurrent.BlockingQueue} fashion and
  * tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ *   <li>{@link #urgentQueue}: urgent mails ({@link 
MailboxExecutor.MailOptions#isUrgent()}) in

Review Comment:
   In this PR we are significantly expanding the number of `Urgent` mails by 
adding operator events. Is it okay that we are now going to delay processing of 
control mails like (reportThrowable, suspend, allActionsCompleted, resume) 
should they arrive after operator events? I am especially concerned about some 
rogue scenarios where coordinator due to whatever bug decides to send a ton of 
such events. 
   
   Would we tradeoff some robustness here? Would it worth to still process 
control mails as LIFO relative to the checkpointing?



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -120,12 +134,12 @@ public interface TaskMailbox {
     Optional<Mail> tryTakeIgnoringDeferrable(int priority);
 
     /**
-     * This method returns the oldest mail from the mailbox (head of queue) or 
blocks until a mail
-     * is available.
+     * This method returns the oldest eligible mail (urgent mails first, see 
{@link #tryTake(int)})

Review Comment:
   We use eligible a lot, what does this term add beyond urgency and priority? 
Are some messages not eligible for processing at all no matter the priority?



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -121,6 +150,10 @@ private Optional<Mail> tryTake(int priority, boolean 
ignoreDeferrable) {
         checkIsMailboxThread();
         checkTakeStateConditions();
 
+        Mail urgentHead = takeUrgentIfPending(priority, ignoreDeferrable);

Review Comment:
   Is it okay to skip lock for this case because the queue head was made 
visible to the thread already?



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
 /**
  * Implementation of {@link TaskMailbox} in a {@link 
java.util.concurrent.BlockingQueue} fashion and
  * tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ *   <li>{@link #urgentQueue}: urgent mails ({@link 
MailboxExecutor.MailOptions#isUrgent()}) in
+ *       arrival order (FIFO), guarded by the lock.
+ *   <li>{@link #queue}: all other mails in arrival order, guarded by the lock.
+ *   <li>{@link #batch}: a bounded round of normal mails that was moved out of 
{@link #queue} by
+ *       {@link #createBatch()}; accessed by the mailbox thread only and 
therefore lock-free.
+ * </ul>
+ *
+ * <p>Mails are served in the order {@code urgentQueue}, {@code batch}, {@code 
queue}. Within each
+ * container the oldest eligible mail goes first, so urgent mails of equal 
priority are FIFO among
+ * themselves, no matter which thread puts them or which method takes them; a 
priority take may skip
+ * an older urgent mail whose priority is below the requested one. Urgent 
mails are the only mails
+ * that may be taken from {@link #tryTakeFromBatch()} although they were not 
part of the batch when
+ * it was created.
  */
 @ThreadSafe
 public class TaskMailboxImpl implements TaskMailbox {
     /** Lock for all concurrent ops. */
     private final ReentrantLock lock = new ReentrantLock();
 
-    /** Internal queue of mails. */
+    /** Internal queue of normal (non-urgent) mails. */

Review Comment:
   !nit, I'd just say "normal/standard" vs "urgent" otherwise "non-urgent" 
feels like definition in reverse. 



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -30,6 +30,15 @@
  * A task mailbox provides read and write access to a mailbox and has a 
lifecycle of open ->
  * (quiesced) -> closed. Mails have a priority that can be used to retrieve 
only relevant letters.
  *
+ * <h3>Priority and urgency</h3>

Review Comment:
   Okay, reading more into this code, I understand that priority is is not a 
"priority" but rather a form of "reentrance protection" for operators 🤔. Maybe 
it should have been called `level/depth` or something like that to start with. 
But obviously not for this PR. Either way, these docs would benefit from a 
diagram putting all terms together, as the concepts here are very ambiguous



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -30,6 +30,15 @@
  * A task mailbox provides read and write access to a mailbox and has a 
lifecycle of open ->
  * (quiesced) -> closed. Mails have a priority that can be used to retrieve 
only relevant letters.
  *
+ * <h3>Priority and urgency</h3>

Review Comment:
   I guess I have mixed up the terms. Yes, 2 queues are `Urgent` and 
`non-Urgent`. And `Urgent` queue is what Rui has introduced in original PR 
(conceptually). 
   
   Could we then simplify this doc to something like:
   ```
   All mails are classified into Standard and Urgent (see 
MailOptions#isUrgent()) and handled as 2 queues. Standard queue is processed 
only when urgent queue is empty. Processing order for both queues is defined by 
Priority (see <link>). Messages with the same priority are processed in FIFO 
order. 
   ```



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -147,20 +179,29 @@ private Optional<Mail> tryTake(int priority, boolean 
ignoreDeferrable) {
         checkIsMailboxThread();
         checkTakeStateConditions();
 
+        Mail urgentHead = takeUrgentIfPending(priority, false);

Review Comment:
   !nit the code here looks a bit branch-y an duplicat-y. The for the logic 
takes first from -> urgent -> then batch -> then standard we could use some 
form of the loop:
   for source in [urgent, batch, standard]:
     tryTake
      if taken -> return



##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -254,15 +327,21 @@ private Mail takeOrNull(Deque<Mail> queue, int priority, 
boolean ignoreDeferrabl
         return null;
     }
 
+    /** Drains all mails in the order in which they would be taken: urgent, 
batch, normal. */
     @Override
     public List<Mail> drain() {
-        List<Mail> drainedMails = new ArrayList<>(batch);
-        batch.clear();

Review Comment:
   Is there any caveat in the fact that we are now clearing batch under lock? 
Does it introduce any extra latency? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to