[
https://issues.apache.org/jira/browse/FLINK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuxin Tan updated FLINK-31634:
------------------------------
Release Note: Hybrid Shuffle supports storing the shuffle data in remote
storage. The remote storage path can be configured with the option
`taskmanager.network.hybrid-shuffle.remote.path`. Hybrid Shuffle uses less
network memory than before by decoupling the memory usage from the number of
parallelisms, improving the stability and ease of use. For detailed
information, please refer to
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/batch/batch_shuffle/#hybrid-shuffle
(was: This introduces a new Hybrid Shuffle mode to support remote storage.
The remote storage path is specified by
`taskmanager.network.hybrid-shuffle.remote.path`. The new mode has resolved
existing issues in the legacy Hybrid Shuffle mode. Firstly, the new mode uses
less required network memory. Secondly, the new mode can store shuffle data in
remote storage when the disk space is not enough, which could avoid
insufficient disk space errors. The new mode is only enabled when
`taskmanager.network.hybrid-shuffle.remote.path` is configured.
Currently, the new Hybrid Shuffle mode is in an experimental phase. In case of
unexpected issues, the new mode can fall back to the legacy mode by setting
`taskmanager.network.hybrid-shuffle.enable-new-mode` to `false`. Once the new
mode is stable, the legacy mode and the fallback option will be removed.)
> FLIP-301: Hybrid Shuffle supports Remote Storage
> ------------------------------------------------
>
> Key: FLINK-31634
> URL: https://issues.apache.org/jira/browse/FLINK-31634
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Affects Versions: 1.18.0
> Reporter: Yuxin Tan
> Assignee: Yuxin Tan
> Priority: Major
> Labels: Umbrella
>
> This is an umbrella ticket for
> [FLIP-301|https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
