rkhachatryan commented on PR #29954:
URL: https://github.com/apache/flink/pull/29954#issuecomment-7051299304

   > My main concern is that we are potentially significantly increasing number 
of "urgent" mails that can overtake the queue, do we know how bounded is this 
new source?
   
   @Izeren good point. Two parts:
   1. Starvation of the input: urgent mails arriving while a batch is consumed 
are served before the rest of the batch, so a sustained stream of them could 
extend it indefinitely. That was already the case in FLINK-35051, but making 
operator events urgent makes it easier to hit. I'll push a commit that caps the 
urgent mails served per batch round at `pending urgent + batch size` (computed 
at `createBatch`). In normal operation urgent mails will still overtake the 
batch, but the round always ends and input gets its turn.
   2. Beyond that, operator events are bounded by the coordinator's send rate 
and are cheap to apply. Starvation requires a sustained event rate above what 
the task can process, and then the backlog grows unboundedly regardless of the 
policy (as it did before FLINK-35051, when events were normal mails).
   
   For reference, how the variants compare (UC = unaligned checkpoint):
   
   | | pre-FLINK-35051 | FLINK-35051 | this PR, uncapped | this PR, with the 
cap |
   |---|---|---|---|---|
   | **UC trigger waits behind…** | | | | |
   | rest of the current batch | yes | no | no | no (yes only once the budget 
is used up) |
   | normal mails queued before it | yes | no | no | no |
   | urgent mails queued before it | n/a | yes, order not guaranteed | yes, 
FIFO | yes, FIFO |
   | one default-action step (if it arrives mid-batch) | yes | no | no | only 
once the budget is used up |
   | **Correctness** | | | | |
   | operator events vs. UC trigger (lost splits) | OK | broken | OK | OK |
   | FIFO among urgent mails | n/a | no | yes | yes |
   | stale gate mail vs. next UC barrier (FLINK-41547) | affected | affected | 
affected | affected |
   | **Control mails** | front of mailbox (LIFO) | urgent, order not guaranteed 
| urgent, FIFO, behind pending urgent mails | same |
   | **Sustained urgent overload** | | | | |
   | input | gets a turn every round | stalls (hard to reach) | stalls | gets a 
turn every round |
   | normal mails while default action is suspended | progress | starve | 
starve | starve |
   | urgent backlog | grows | grows | grows | grows |
   
   The one-step wait is the price for input getting a turn under overload.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to