rkhachatryan commented on code in PR #29954:
URL: https://github.com/apache/flink/pull/29954#discussion_r4648043553
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -155,191 +179,134 @@ private Optional<Mail> tryTake(int priority, boolean
ignoreDeferrable) {
checkIsMailboxThread();
checkTakeStateConditions();
- moveUrgentMailsToBatchIfNeeded(true);
-
+ Mail urgentHead = takeUrgentIfPending(priority, false);
+ if (urgentHead != null) {
+ return urgentHead;
+ }
Mail head = takeOrNull(batch, priority, false);
if (head != null) {
return head;
}
final ReentrantLock lock = this.lock;
lock.lockInterruptibly();
try {
- Mail headMail;
- while ((headMail = takeOrNull(queue, priority, false)) == null) {
+ while (true) {
+ Mail urgent = takeUrgentLocked(priority, false);
+ if (urgent != null) {
+ return urgent;
+ }
+ Mail mail = takeNormalLocked(priority, false);
+ if (mail != null) {
+ return mail;
+ }
// to ease debugging
notEmpty.await(1, TimeUnit.SECONDS);
}
- updateNewMailFlags();
- return headMail;
} finally {
lock.unlock();
}
}
- private void updateNewMailFlags() {
- Mail peek = queue.peek();
- if (peek != null) {
- hasNewMail = true;
- hasNewUrgentMail = peek.getMailOptions().isUrgent();
- } else {
- hasNewMail = false;
- hasNewUrgentMail = false;
- }
- }
-
//
------------------------------------------------------------------------------------------------------------------
@Override
public boolean createBatch() {
- moveUrgentMailsToBatchIfNeeded(false);
- return !batch.isEmpty();
- }
-
- /**
- * Try to create a batch of mails that can be taken with {@link
#tryTakeFromBatch()}. The batch
- * does not affect {@link #tryTake(int)} and {@link #take(int)}; that is,
they return the same
- * mails even if no batch had been created.
- *
- * <p>If a batch is not completely consumed by {@link
#tryTakeFromBatch()}, its elements are
- * carried over to the new batch.
- *
- * <p>Must be called from the mailbox thread ({@link #isMailboxThread()}.
- *
- * <p>To ensure that urgent mails are executed in FIFO order, moving mails
from queue to batch
- * only happens when batch doesn't have urgent mails. In addition, in
order to reduce the
- * frequent acquisition of locks caused by moving, moving is not required
if batch is not empty
- * and no new urgent mails.
- *
- * <pre>{@code
- * All received mails (from head to tail):
- * Urgent mails: MailA -> MailB -> MailC -> MailD -> MailE -> MailF
- * Non-urgent mails: Mail1 -> Mail2 -> Mail3 -> Mail4 -> Mail5 -> Mail6
-> Mail7
- *
- * All received mails (The actual storage structure):
- * queue: UrgentMails and NonUrgentMails (UrgentMails is in reverse
order)
- * MailF - MailE - MailD - Mail6 - Mail7 (the second batch)
- * batch: UrgentMails and NonUrgentMails (Both are FIFO)
- * MailA - MailB - MailC - Mail1 - Mail2 - Mail3 - Mail4 - Mail5 (the
first batch)
- *
- * To ensure that urgent mails are executed in FIFO order, moving mails
from queue to batch
- * only happens when batch doesn't have urgent mails. For examples, the
moving will happen when
- * MailA, MailB and MailC are taken from batch:
- *
- * Step1 : moving all urgent mails into the head of batch:
- * queue: Mail6 - Mail7 - Mail8
- * batch: MailE - MailF - MailG - Mail1 - Mail2 - Mail3 - Mail4
- * Step2 : moving all non-urgent mails into the tail of batch:
- * queue:
- * batch: MailE - MailF - MailG - Mail1 - Mail2 - Mail3 - Mail4 - Mail5
- Mail6 - Mail7
- * }</pre>
- */
- private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
checkIsMailboxThread();
- Mail peek = batch.peek();
- if (peek != null) {
- if (peek.getMailOptions().isUrgent()) {
- // To ensure that urgent mails are executed in FIFO order,
moving mails from queue
- // to batch only happens when batch doesn't have urgent mails.
- return;
- } else if (!hasNewUrgentMail) {
- // In order to reduce the frequent lock acquisition caused by
movement, if the batch
- // is not empty and there are no new urgent emails, there is
no need to move.
- return;
- }
- } else {
- if (onlyMoveUrgentMails && !hasNewUrgentMail) {
- return;
- }
- if (!hasNewMail) {
- // Both batch and queue are empty
- return;
- }
- }
-
- final ReentrantLock lock = this.lock;
- lock.lock();
- try {
- Mail mail;
- while ((mail = queue.pollFirst()) != null) {
- if (mail.getMailOptions().isUrgent()) {
- batch.addFirst(mail);
- } else {
- if (onlyMoveUrgentMails) {
- // Put non-urgent mail back into the queue, and stop
the loop
- queue.addFirst(mail);
- break;
- } else {
- batch.addLast(mail);
- }
- }
+ if (hasNewMail) {
+ final ReentrantLock lock = this.lock;
+ lock.lock();
+ try {
+ batch.addAll(queue);
+ queue.clear();
+ hasNewMail = false;
+ } finally {
+ lock.unlock();
}
- hasNewUrgentMail = false;
- hasNewMail = !queue.isEmpty();
- } finally {
- lock.unlock();
}
+ // pending urgent mails are served by tryTakeFromBatch, too
+ return !batch.isEmpty() || hasNewUrgentMail;
}
@Override
public Optional<Mail> tryTakeFromBatch() {
checkIsMailboxThread();
checkTakeStateConditions();
- moveUrgentMailsToBatchIfNeeded(true);
+ Mail urgent = takeUrgentIfPending(Integer.MIN_VALUE, false);
+ if (urgent != null) {
+ return Optional.of(urgent);
Review Comment:
Will push a commit bounding the number of urgent mails served per batch
round, with a test (`testUrgentMailsCannotExtendBatchUnboundedly`).
##########
flink-core/src/main/java/org/apache/flink/api/common/operators/MailboxExecutor.java:
##########
@@ -89,21 +89,35 @@ public interface MailboxExecutor {
/** Extra options to configure enqueued mails. */
@PublicEvolving
interface MailOptions {
- static MailOptions options() {
- return MailOptionsImpl.DEFAULT;
- }
/**
- * Mark this mail as deferrable.
- *
- * <p>Runtime can decide to defer execution of deferrable mails. For
example, to unblock
Review Comment:
That came from re-applying FLINK-35051 as-is. Will restore the original
order and javadoc to minimize the diff.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/MailboxProcessor.java:
##########
@@ -330,15 +330,17 @@ private void sendPoisonMail(RunnableWithException mail) {
}
/**
- * Sends the given <code>mail</code> using {@link
TaskMailbox#putFirst(Mail)} . Intended use is
- * to control this <code>MailboxProcessor</code>; no interaction with
tasks should be performed;
+ * Sends the given <code>mail</code> with urgent. Intended use is to
control this <code>
+ * MailboxProcessor</code>; no interaction with tasks should be performed.
The mail is served
+ * before all non-urgent mails, but after urgent mails that are already
pending (FIFO).
*/
private void sendControlMail(
RunnableWithException mail, String descriptionFormat, Object...
descriptionArgs) {
- mailbox.putFirst(
+ mailbox.put(
Review Comment:
Control mails are urgent with `MAX_PRIORITY`: they overtake all normal mails
and pass any priority filter, and are FIFO among urgent mails (including other
control mails). So yes, it's strict FIFO now, which also removes the race you
mention.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -120,12 +134,12 @@ public interface TaskMailbox {
Optional<Mail> tryTakeIgnoringDeferrable(int priority);
/**
- * This method returns the oldest mail from the mailbox (head of queue) or
blocks until a mail
- * is available.
+ * This method returns the oldest eligible mail (urgent mails first, see
{@link #tryTake(int)})
Review Comment:
Will remove the term and spell it out ("priority at least the given one").
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
/**
* Implementation of {@link TaskMailbox} in a {@link
java.util.concurrent.BlockingQueue} fashion and
* tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ * <li>{@link #urgentQueue}: urgent mails ({@link
MailboxExecutor.MailOptions#isUrgent()}) in
Review Comment:
A control mail waits only for the urgent mails already pending when it's
put. These drain unless the coordinator keeps overloading the task (see my
reply to your review comment), in which case the job is in trouble anyway. In
return, control mails are now FIFO among themselves, which I find more
predictable than the previous LIFO from `putFirst`.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
/**
* Implementation of {@link TaskMailbox} in a {@link
java.util.concurrent.BlockingQueue} fashion and
* tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ * <li>{@link #urgentQueue}: urgent mails ({@link
MailboxExecutor.MailOptions#isUrgent()}) in
+ * arrival order (FIFO), guarded by the lock.
+ * <li>{@link #queue}: all other mails in arrival order, guarded by the lock.
+ * <li>{@link #batch}: a bounded round of normal mails that was moved out of
{@link #queue} by
+ * {@link #createBatch()}; accessed by the mailbox thread only and
therefore lock-free.
+ * </ul>
+ *
+ * <p>Mails are served in the order {@code urgentQueue}, {@code batch}, {@code
queue}. Within each
+ * container the oldest eligible mail goes first, so urgent mails of equal
priority are FIFO among
+ * themselves, no matter which thread puts them or which method takes them; a
priority take may skip
+ * an older urgent mail whose priority is below the requested one. Urgent
mails are the only mails
+ * that may be taken from {@link #tryTakeFromBatch()} although they were not
part of the batch when
+ * it was created.
*/
@ThreadSafe
public class TaskMailboxImpl implements TaskMailbox {
/** Lock for all concurrent ops. */
private final ReentrantLock lock = new ReentrantLock();
- /** Internal queue of mails. */
+ /** Internal queue of normal (non-urgent) mails. */
Review Comment:
Will do, "normal" everywhere.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -121,6 +150,10 @@ private Optional<Mail> tryTake(int priority, boolean
ignoreDeferrable) {
checkIsMailboxThread();
checkTakeStateConditions();
+ Mail urgentHead = takeUrgentIfPending(priority, ignoreDeferrable);
Review Comment:
Yes, it's safe. If an urgent mail is put after the volatile read, the locked
path in `tryTake` checks the urgent queue first again. If we return a batch
mail instead, that take simply linearizes before the put.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -30,6 +30,15 @@
* A task mailbox provides read and write access to a mailbox and has a
lifecycle of open ->
* (quiesced) -> closed. Mails have a priority that can be used to retrieve
only relevant letters.
*
+ * <h3>Priority and urgency</h3>
Review Comment:
Good idea. I'll push a rewrite along those lines: urgent vs. normal, FIFO
within each, urgent first; priority only filters. It'll include a small diagram
too.
Agreed on priority being a reentrance level; renaming is out of scope here,
but the updated doc will explain it in those terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]