[
https://issues.apache.org/jira/browse/FLINK-33077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33077:
-----------------------------------
Labels: pull-request-available (was: )
> Minimize the risk of hard back-pressure with buffer debloating enabled
> ----------------------------------------------------------------------
>
> Key: FLINK-33077
> URL: https://issues.apache.org/jira/browse/FLINK-33077
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: Stefan Richter
> Assignee: Stefan Richter
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> {*}Problem{*}:
> Buffer debloating sets buffer size to {{256}} bytes because of back-pressure.
> Such small buffers might not be enough to emit the processing results of a
> single record. The task thread would request new buffers, and often block.
> That results in significant checkpoint delays (up to minutes instead of
> seconds).
> Adding more overdraft buffers helps, but depends on the job DoP
> Raising {{taskmanager.memory.min-segment-size}} from {{256}} helps, but
> depends on the multiplication factor of the operator.
> {*}Solution{*}:
> * Ignore Buffer Debloater hints and extend the buffer if possible - when
> this prevents emitting an output record fully AND this is the last available
> buffer.
> * Prevent the subsequent flush of the buffer so that more output records can
> be emitted (flatMap-like and join operators)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
