[
https://issues.apache.org/jira/browse/FLINK-30469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-30469.
--------------------------------
Fix Version/s: 1.17.0
Release Note:
The default value of `taskmanager.memory.network.max` has changed from `1g` to
`Long#MAX_VALUE`, to reduce the number of config options user needs to tune
when trying to increase the network memory size. This may affect the
performance when this option is not explicitly configured, due to potential
changes of network memory size, and heap and managed memory sizes when the
total memory size is fixed. To go back to the previous behavior, user can
explicitly configure this option to the previous default value `1g`.
A threshold is introduced for controlling the number of required buffers among
all buffers needed for reading data from upstream tasks. Reducing the number of
required buffers helps reduces the chance of failures due to insufficient
network buffers, at the price of potential performance impact. By default, the
number of required buffers is only reduced for batch workloads, while stay
unchanged for streaming workloads. This can be tuned via
`taskmanager.network.memory.read-buffer.required-per-gate.max`. See the
description of the config option for more details.
Resolution: Done
> FLIP-266: Simplify network memory configurations for TaskManager
> ----------------------------------------------------------------
>
> Key: FLINK-30469
> URL: https://issues.apache.org/jira/browse/FLINK-30469
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.17.0
> Reporter: Yuxin Tan
> Assignee: Yuxin Tan
> Priority: Major
> Fix For: 1.17.0
>
>
> When using Flink, users may encounter the following issues that affect
> usability.
> 1. The job may fail with an "Insufficient number of network buffers"
> exception.
> 2. Flink network memory size adjustment is complex.
> When encountering these issues, users can solve some problems by adding or
> adjusting parameters. However, multiple memory config options should be
> changed. The config option adjustment requires understanding the detailed
> internal implementation, which is impractical for most users.
> To resolve the issues, we propose some improvement solutions. For more
> details see
> [FLIP-266|https://cwiki.apache.org/confluence/display/FLINK/FLIP-266%3A+Simplify+network+memory+configurations+for+TaskManager].
> This is the umbrella ticket to track all the changes of this feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
