[
https://issues.apache.org/jira/browse/FLINK-31634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuxin Tan updated FLINK-31634:
------------------------------
Release Note:
This introduces a new Hybrid Shuffle mode to support remote storage. The remote
storage path is specified by `taskmanager.network.hybrid-shuffle.remote.path`.
The new Hybrid Shuffle mode has resolved existing issues in the legacy mode.
Firstly, the new mode uses less required network memory. Secondly, the new mode
can store shuffle data in remote storage when the disk space is not enough,
which could avoid insufficient disk space errors. The new mode is only enabled
when `taskmanager.network.hybrid-shuffle.remote.path` is configured.
Currently, the new Hybrid Shuffle mode is in an experimental phase. In case of
unexpected issues, the new mode can fallback to the legacy mode by setting
`taskmanager.network.hybrid-shuffle.enable-new-mode` to `false`. Once the new
mode is stable, the legacy mode and the fallback option will be removed.
was:
This introduces a new Hybrid Shuffle mode to support remote storage. The remote
storage path is specified by `taskmanager.network.hybrid-shuffle.remote.path`.
The new Hybrid Shuffle mode has resolved existing issues in the legacy mode.
Firstly, the new mode uses less required network memory. Secondly, the new mode
can store shuffle data in remote storage when the disk space is not enough,
which could avoid insufficient disk space errors. The new mode is only enabled
when `taskmanager.network.hybrid-shuffle.remote.path` is configured.
Currently, the new Hybrid Shuffle mode is in an experimental phase. In case of
unexpected issues, the new mode can fallback to the legacy mode by setting
`taskmanager.network.hybrid-shuffle.enable-new-mode` to `false`. Once the new
mode is stable, the legacy mode and its option will be removed.
> FLIP-301: Hybrid Shuffle supports Remote Storage
> ------------------------------------------------
>
> Key: FLINK-31634
> URL: https://issues.apache.org/jira/browse/FLINK-31634
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Network
> Affects Versions: 1.18.0
> Reporter: Yuxin Tan
> Assignee: Yuxin Tan
> Priority: Major
> Labels: Umbrella
>
> This is an umbrella ticket for
> [FLIP-301|https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
