This is an automated email from the ASF dual-hosted git repository. pnowojski pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git
commit a17d824e1797f3424cea9c12e0a9bc9a4d654741 Author: Piotr Nowojski <[email protected]> AuthorDate: Tue Jun 21 10:14:09 2022 +0200 [FLINK-26762][docs] Document overdraft buffers --- .../content.zh/docs/deployment/memory/network_mem_tuning.md | 13 +++++++++++++ docs/content/docs/deployment/memory/network_mem_tuning.md | 13 +++++++++++++ 2 files changed, 26 insertions(+) diff --git a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md index cccba3bff4b..8adc281e225 100644 --- a/docs/content.zh/docs/deployment/memory/network_mem_tuning.md +++ b/docs/content.zh/docs/deployment/memory/network_mem_tuning.md @@ -118,6 +118,19 @@ Flink 有多个本地缓冲区池 —— 每个输出和输入流对应一个。 不同于输入缓冲区池，这里配置的独占缓冲区和流动缓冲区只被当作推荐值。如果没有足够的缓冲区，每个输出 subpartition 可以只使用一个独占缓冲区而没有流动缓冲区。 +#### Overdraft buffers + +For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers. +Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output. +This can happen in situations like: +- Serializing very large records, that do not fit into a single network buffer. +- Flat Map like operator, that produces many output records per single input record. +- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`). + +Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints +from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can +make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers. + ## 缓冲区的数量 独占缓冲区和流动缓冲区的默认配置应该足以应对最大吞吐。如果想要最小化缓冲数据量，那么可以将独占缓冲区设置为 `0`，同时减小内存段的大小。 diff --git a/docs/content/docs/deployment/memory/network_mem_tuning.md b/docs/content/docs/deployment/memory/network_mem_tuning.md index 9c537517b89..a29cdfa133d 100644 --- a/docs/content/docs/deployment/memory/network_mem_tuning.md +++ b/docs/content/docs/deployment/memory/network_mem_tuning.md @@ -120,6 +120,19 @@ In order to avoid excessive data skew, the number of buffers for each subpartiti Unlike the input buffer pool, the configured amount of exclusive buffers and floating buffers is only treated as recommended values. If there are not enough buffers available, Flink can make progress with only a single exclusive buffer per output subpartition and zero floating buffers. +#### Overdraft buffers + +For each output subtask can also request up to `taskmanager.network.memory.max-overdraft-buffers-per-gate` (by default 5) extra overdraft buffers. +Those buffers are only used, if despite presence of a backpressure, Flink can not stop producing more records to the output. +This can happen in situations like: +- Serializing very large records, that do not fit into a single network buffer. +- Flat Map like operator, that produces many output records per single input record. +- Operators that output many records either periodically or on a reaction to some event (for example `WindowOperator`). + +Without overdraft buffers in such situations Flink subtask thread would block on the backpressure, preventing for example unaligned checkpoints +from being triggered. To mitigate this, the overdraft buffers concept has been added. Those buffers are strictly optional and Flink can +make progress even if the Task Manager doesn't have any spare buffers in the global pool to be used as overdraft buffers. + ## The number of in-flight buffers The default settings for exclusive buffers and floating buffers should be sufficient for the maximum throughput. If the minimum of in-flight data needs to be set, the exclusive buffers can be set to `0` and the memory segment size can be decreased.
