Stefan Richter created FLINK-33077: -------------------------------------- Summary: Minimize the risk of hard back-pressure with buffer debloating enabled Key: FLINK-33077 URL: https://issues.apache.org/jira/browse/FLINK-33077 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing Reporter: Stefan Richter Assignee: Stefan Richter Fix For: 1.18.0
{*}Problem{*}: Buffer debloating sets buffer size to {{256}} bytes because of back-pressure. Such small buffers might not be enough to emit the processing results of a single record. The task thread would request new buffers, and often block. That results in significant checkpoint delays (up to minutes instead of seconds). Adding more overdraft buffers helps, but depends on the job DoP Raising {{taskmanager.memory.min-segment-size}} from {{256}} helps, but depends on the multiplication factor of the operator. {*}Solution{*}: * Ignore Buffer Debloater hints and extend the buffer if possible - when this prevents emitting an output record fully AND this is the last available buffer. * Prevent the subsequent flush of the buffer so that more output records can be emitted (flatMap-like and join operators) -- This message was sent by Atlassian Jira (v8.20.10#820010)