TanYuxin-tyx commented on code in PR #21843:
URL: https://github.com/apache/flink/pull/21843#discussion_r1100028365
##########
docs/content/docs/ops/batch/batch_shuffle.md:
##########
@@ -76,8 +76,9 @@ The memory usage of `mmap` is not accounted for by configured
memory limits, but
`Sort Shuffle` is another blocking shuffle implementation introduced in
version 1.13 and it becomes the default blocking shuffle implementation in
1.15. Different from `Hash Shuffle`, `Sort Shuffle` writes only one file for
each result partition. When the result partition is read by multiple downstream
tasks concurrently, the data file is opened only once and shared by all
readers. As a result, the cluster uses fewer resources like inode and file
descriptors, which improves stability. Furthermore, by writing fewer files and
making a best effort to read data sequentially, `Sort Shuffle` can achieve
better performance than `Hash Shuffle`, especially on HDD. Additionally, `Sort
Shuffle` uses extra managed memory as data reading buffer and does not rely on
`sendfile` or `mmap` mechanism, thus it also works well with [SSL]({{< ref
"docs/deployment/security/security-ssl" >}}). Please refer to
[FLINK-19582](https://issues.apache.org/jira/browse/FLINK-19582) and
[FLINK-19614](https://issues
.apache.org/jira/browse/FLINK-19614) for more details about `Sort Shuffle`.
Here are some config options that might need adjustment when using sort
blocking shuffle:
-- [taskmanager.network.sort-shuffle.min-buffers]({{< ref
"docs/deployment/config" >}}#taskmanager-network-sort-shuffle-min-buffers):
Config option to control data writing buffer size. For large scale jobs, you
may need to increase this value, usually, several hundreds of megabytes memory
is enough. Because this memory is allocated from network memory, to increase
this value, you may also need to increase the total network memory by adjusting
[taskmanager.memory.network.fraction]({{< ref "docs/deployment/config"
>}}#taskmanager-memory-network-fraction), [taskmanager.memory.network.min]({{<
ref "docs/deployment/config" >}}#taskmanager-memory-network-min) and
[taskmanager.memory.network.max]({{< ref "docs/deployment/config"
>}}#taskmanager-memory-network-max) to avoid the potential "Insufficient number
of network buffers" error.
+- [taskmanager.network.sort-shuffle.min-buffers]({{< ref
"docs/deployment/config" >}}#taskmanager-network-sort-shuffle-min-buffers):
Config option to control data writing buffer size. For large scale jobs, you
may need to increase this value, usually, several hundreds of megabytes memory
is enough. Because this memory is allocated from network memory, to increase
this value, you may also need to increase the total network memory by adjusting
[taskmanager.memory.network.fraction]({{< ref "docs/deployment/config"
>}}#taskmanager-memory-network-fraction), [taskmanager.memory.network.min]({{<
ref "docs/deployment/config" >}}#taskmanager-memory-network-min) to avoid the
potential "Insufficient number of network buffers" error.
- [taskmanager.memory.framework.off-heap.batch-shuffle.size]({{< ref
"docs/deployment/config"
>}}#taskmanager-memory-framework-off-heap-batch-shuffle-size): Config option to
control data reading buffer size. For large scale jobs, you may need to
increase this value, usually, several hundreds of megabytes memory is enough.
Because this memory is cut from the framework off-heap memory, to increase this
value, you need also to increase the total framework off-heap memory by
adjusting [taskmanager.memory.framework.off-heap.size]({{< ref
"docs/deployment/config" >}}#taskmanager-memory-framework-off-heap-size) to
avoid the potential direct memory OOM error.
+- [taskmanager.memory.network.max]({{< ref "docs/deployment/config"
>}}#taskmanager-memory-network-max): Config option to control the maximum total
network memory. Since 1.17, this option is set as Long.MAX_VALUE by default,
which means that the maximum network memory size is no longer limited. If you
still want to limit the maximum network memory, set the config to a new value,
for example, 1g.
Review Comment:
I have reverted the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
