pnowojski commented on code in PR #20038: URL: https://github.com/apache/flink/pull/20038#discussion_r902546911
########## docs/content/docs/deployment/memory/network_mem_tuning.md: ########## @@ -120,6 +120,19 @@ In order to avoid excessive data skew, the number of buffers for each subpartiti Unlike the input buffer pool, the configured amount of exclusive buffers and floating buffers is only treated as recommended values. If there are not enough buffers available, Flink can make progress with only a single exclusive buffer per output subpartition and zero floating buffers. +#### Overdraft buffers + +For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers. +Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output. +This can happen in situations like: +- Serializing very large records, that do not fit into a single network buffer. +- Flat Map like operator, that produces many output records per single input record. +- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`). + +Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints +from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can +make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers. Review Comment: This is what I meant, but I see why it's confusing. I will rephrase this. ########## docs/content/docs/deployment/memory/network_mem_tuning.md: ########## @@ -120,6 +120,19 @@ In order to avoid excessive data skew, the number of buffers for each subpartiti Unlike the input buffer pool, the configured amount of exclusive buffers and floating buffers is only treated as recommended values. If there are not enough buffers available, Flink can make progress with only a single exclusive buffer per output subpartition and zero floating buffers. +#### Overdraft buffers + +For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers. +Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output. Review Comment: Hmmm, I've rephrased it to this: > Those buffers are only used, if the subtask is backpressured > by downstream subtasks but the subtask can not gracefully pause its current process. This can happen > in situations like: From your proposal I don't like: > the subtask still needs to produce more records to the output. as it can be easily confused with regular processing of records from the input and producing a single output record. ########## docs/content/docs/deployment/memory/network_mem_tuning.md: ########## @@ -120,6 +120,19 @@ In order to avoid excessive data skew, the number of buffers for each subpartiti Unlike the input buffer pool, the configured amount of exclusive buffers and floating buffers is only treated as recommended values. If there are not enough buffers available, Flink can make progress with only a single exclusive buffer per output subpartition and zero floating buffers. +#### Overdraft buffers + +For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers. +Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output. +This can happen in situations like: +- Serializing very large records, that do not fit into a single network buffer. +- Flat Map like operator, that produces many output records per single input record. +- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`). + +Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints +from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can Review Comment: 👍 I've rephrased it to: > Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints from completing. ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
