Izeren commented on code in PR #29954:
URL: https://github.com/apache/flink/pull/29954#discussion_r4612202215
##########
flink-core/src/main/java/org/apache/flink/api/common/operators/MailboxExecutor.java:
##########
@@ -89,21 +89,35 @@ public interface MailboxExecutor {
/** Extra options to configure enqueued mails. */
@PublicEvolving
interface MailOptions {
- static MailOptions options() {
- return MailOptionsImpl.DEFAULT;
- }
/**
- * Mark this mail as deferrable.
- *
- * <p>Runtime can decide to defer execution of deferrable mails. For
example, to unblock
Review Comment:
!nit you have changed formatting here.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/MailboxProcessor.java:
##########
@@ -330,15 +330,17 @@ private void sendPoisonMail(RunnableWithException mail) {
}
/**
- * Sends the given <code>mail</code> using {@link
TaskMailbox#putFirst(Mail)} . Intended use is
- * to control this <code>MailboxProcessor</code>; no interaction with
tasks should be performed;
+ * Sends the given <code>mail</code> with urgent. Intended use is to
control this <code>
+ * MailboxProcessor</code>; no interaction with tasks should be performed.
The mail is served
+ * before all non-urgent mails, but after urgent mails that are already
pending (FIFO).
*/
private void sendControlMail(
RunnableWithException mail, String descriptionFormat, Object...
descriptionArgs) {
- mailbox.putFirst(
+ mailbox.put(
Review Comment:
Are we no longer treating control mails as top priority? Or is it top
priority except for urgent?
As I understand, the original code (before Rui's changes) could actually
have control messages overtaking each other in case 2 threads were racing to
put control mail before mailbox thread was waken up. Do we change the logic to
strict FIFO here and place new message after any other that also has
MAX_PRIORITY?
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
/**
* Implementation of {@link TaskMailbox} in a {@link
java.util.concurrent.BlockingQueue} fashion and
* tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ * <li>{@link #urgentQueue}: urgent mails ({@link
MailboxExecutor.MailOptions#isUrgent()}) in
Review Comment:
In this PR we are significantly expanding the number of `Urgent` mails by
adding operator events. Is it okay that we are now going to delay processing of
control mails like (reportThrowable, suspend, allActionsCompleted, resume)
should they arrive after operator events? I am especially concerned about some
rogue scenarios where coordinator due to whatever bug decides to send a ton of
such events.
Would we tradeoff some robustness here? Would it worth to still process
control mails as LIFO relative to the checkpointing?
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -120,12 +134,12 @@ public interface TaskMailbox {
Optional<Mail> tryTakeIgnoringDeferrable(int priority);
/**
- * This method returns the oldest mail from the mailbox (head of queue) or
blocks until a mail
- * is available.
+ * This method returns the oldest eligible mail (urgent mails first, see
{@link #tryTake(int)})
Review Comment:
We use eligible a lot, what does this term add beyond urgency and priority?
Are some messages not eligible for processing at all no matter the priority?
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -121,6 +150,10 @@ private Optional<Mail> tryTake(int priority, boolean
ignoreDeferrable) {
checkIsMailboxThread();
checkTakeStateConditions();
+ Mail urgentHead = takeUrgentIfPending(priority, ignoreDeferrable);
Review Comment:
Is it okay to skip lock for this case because the queue head was made
visible to the thread already?
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -43,16 +44,37 @@
/**
* Implementation of {@link TaskMailbox} in a {@link
java.util.concurrent.BlockingQueue} fashion and
* tailored towards our use case with multiple writers and single reader.
+ *
+ * <p>Mails are stored in three containers:
+ *
+ * <ul>
+ * <li>{@link #urgentQueue}: urgent mails ({@link
MailboxExecutor.MailOptions#isUrgent()}) in
+ * arrival order (FIFO), guarded by the lock.
+ * <li>{@link #queue}: all other mails in arrival order, guarded by the lock.
+ * <li>{@link #batch}: a bounded round of normal mails that was moved out of
{@link #queue} by
+ * {@link #createBatch()}; accessed by the mailbox thread only and
therefore lock-free.
+ * </ul>
+ *
+ * <p>Mails are served in the order {@code urgentQueue}, {@code batch}, {@code
queue}. Within each
+ * container the oldest eligible mail goes first, so urgent mails of equal
priority are FIFO among
+ * themselves, no matter which thread puts them or which method takes them; a
priority take may skip
+ * an older urgent mail whose priority is below the requested one. Urgent
mails are the only mails
+ * that may be taken from {@link #tryTakeFromBatch()} although they were not
part of the batch when
+ * it was created.
*/
@ThreadSafe
public class TaskMailboxImpl implements TaskMailbox {
/** Lock for all concurrent ops. */
private final ReentrantLock lock = new ReentrantLock();
- /** Internal queue of mails. */
+ /** Internal queue of normal (non-urgent) mails. */
Review Comment:
!nit, I'd just say "normal/standard" vs "urgent" otherwise "non-urgent"
feels like definition in reverse.
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -30,6 +30,15 @@
* A task mailbox provides read and write access to a mailbox and has a
lifecycle of open ->
* (quiesced) -> closed. Mails have a priority that can be used to retrieve
only relevant letters.
*
+ * <h3>Priority and urgency</h3>
Review Comment:
Okay, reading more into this code, I understand that priority is is not a
"priority" but rather a form of "reentrance protection" for operators 🤔. Maybe
it should have been called `level/depth` or something like that to start with.
But obviously not for this PR. Either way, these docs would benefit from a
diagram putting all terms together, as the concepts here are very ambiguous
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailbox.java:
##########
@@ -30,6 +30,15 @@
* A task mailbox provides read and write access to a mailbox and has a
lifecycle of open ->
* (quiesced) -> closed. Mails have a priority that can be used to retrieve
only relevant letters.
*
+ * <h3>Priority and urgency</h3>
Review Comment:
I guess I have mixed up the terms. Yes, 2 queues are `Urgent` and
`non-Urgent`. And `Urgent` queue is what Rui has introduced in original PR
(conceptually).
Could we then simplify this doc to something like:
```
All mails are classified into Standard and Urgent (see
MailOptions#isUrgent()) and handled as 2 queues. Standard queue is processed
only when urgent queue is empty. Processing order for both queues is defined by
Priority (see <link>). Messages with the same priority are processed in FIFO
order.
```
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -147,20 +179,29 @@ private Optional<Mail> tryTake(int priority, boolean
ignoreDeferrable) {
checkIsMailboxThread();
checkTakeStateConditions();
+ Mail urgentHead = takeUrgentIfPending(priority, false);
Review Comment:
!nit the code here looks a bit branch-y an duplicat-y. The for the logic
takes first from -> urgent -> then batch -> then standard we could use some
form of the loop:
for source in [urgent, batch, standard]:
tryTake
if taken -> return
##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -254,15 +327,21 @@ private Mail takeOrNull(Deque<Mail> queue, int priority,
boolean ignoreDeferrabl
return null;
}
+ /** Drains all mails in the order in which they would be taken: urgent,
batch, normal. */
@Override
public List<Mail> drain() {
- List<Mail> drainedMails = new ArrayList<>(batch);
- batch.clear();
Review Comment:
Is there any caveat in the fact that we are now clearing batch under lock?
Does it introduce any extra latency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]