Stefan Richter created FLINK-33077:
--------------------------------------
Summary: Minimize the risk of hard back-pressure with buffer
debloating enabled
Key: FLINK-33077
URL: https://issues.apache.org/jira/browse/FLINK-33077
Project: Flink
Issue Type: Improvement
Components: Runtime / Checkpointing
Reporter: Stefan Richter
Assignee: Stefan Richter
Fix For: 1.18.0
{*}Problem{*}:
Buffer debloating sets buffer size to {{256}} bytes because of back-pressure.
Such small buffers might not be enough to emit the processing results of a
single record. The task thread would request new buffers, and often block.
That results in significant checkpoint delays (up to minutes instead of
seconds).
Adding more overdraft buffers helps, but depends on the job DoP
Raising {{taskmanager.memory.min-segment-size}} from {{256}} helps, but depends
on the multiplication factor of the operator.
{*}Solution{*}:
* Ignore Buffer Debloater hints and extend the buffer if possible - when this
prevents emitting an output record fully AND this is the last available buffer.
* Prevent the subsequent flush of the buffer so that more output records can
be emitted (flatMap-like and join operators)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
