This is an automated email from the ASF dual-hosted git repository. xtsong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/flink.git
commit b9d8bb11cf6fcfab3a63e5987a93cacd31804e9c Author: sxnan <[email protected]> AuthorDate: Mon May 27 15:35:21 2024 +0800 [FLINK-35461][config] Deprecate options related to legacy hybrid shuffle --- .../generated/all_taskmanager_network_section.html | 18 ------------- .../netty_shuffle_environment_configuration.html | 18 ------------- .../NettyShuffleEnvironmentOptions.java | 30 +++++++++++++++++----- 3 files changed, 23 insertions(+), 43 deletions(-) diff --git a/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html b/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html index fae3bc786da..4078de37795 100644 --- a/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html +++ b/docs/layouts/shortcodes/generated/all_taskmanager_network_section.html @@ -26,36 +26,18 @@ <td>Boolean</td> <td>Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.enable-new-mode</h5></td> - <td style="word-wrap: break-word;">true</td> - <td>Boolean</td> - <td>The option is used to enable the new mode of hybrid shuffle, which has resolved existing issues in the legacy mode. First, the new mode uses less required network memory. Second, the new mode can store shuffle data in remote storage when the disk space is not enough, which could avoid insufficient disk space errors and is only supported when taskmanager.network.hybrid-shuffle.remote.path is configured. The new mode is currently in an experimental phase. It can be set to f [...] - </tr> <tr> <td><h5>taskmanager.network.hybrid-shuffle.memory-decoupling.enabled</h5></td> <td style="word-wrap: break-word;">false</td> <td>Boolean</td> <td>The option is used to make the memory that is used by hybrid shuffle decoupled from the complexity of the job topology and the number of tasks on the task manager. It significantly reduces the chance of the "Insufficient number of network buffers" exception, while the workloads may suffer performance reduction silently.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.num-retained-in-memory-regions-max</h5></td> - <td style="word-wrap: break-word;">1048576</td> - <td>Long</td> - <td>Controls the max number of hybrid retained regions in memory. Note: This option will be ignored if taskmanager.network.hybrid-shuffle.enable-new-mode is set true. </td> - </tr> <tr> <td><h5>taskmanager.network.hybrid-shuffle.remote.path</h5></td> <td style="word-wrap: break-word;">(none)</td> <td>String</td> <td>The option is used to configure the base path of remote storage for hybrid shuffle. The shuffle data will be stored in remote storage when the disk space is not enough. Note: If the option is configured and taskmanager.network.hybrid-shuffle.enable-new-mode is false, this option will be ignored. If the option is not configured and taskmanager.network.hybrid-shuffle.enable-new-mode is true, the remote storage will be disabled.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.spill-index-region-group-size</h5></td> - <td style="word-wrap: break-word;">1024</td> - <td>Integer</td> - <td>Controls the region group size(in bytes) of hybrid spilled file data index. Note: This option will be ignored if taskmanager.network.hybrid-shuffle.enable-new-mode is set true.</td> - </tr> <tr> <td><h5>taskmanager.network.max-num-tcp-connections</h5></td> <td style="word-wrap: break-word;">1</td> diff --git a/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html b/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html index 713bc705663..df099186eea 100644 --- a/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html +++ b/docs/layouts/shortcodes/generated/netty_shuffle_environment_configuration.html @@ -44,36 +44,18 @@ <td>Boolean</td> <td>Boolean flag to enable/disable more detailed metrics about inbound/outbound network queue lengths.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.enable-new-mode</h5></td> - <td style="word-wrap: break-word;">true</td> - <td>Boolean</td> - <td>The option is used to enable the new mode of hybrid shuffle, which has resolved existing issues in the legacy mode. First, the new mode uses less required network memory. Second, the new mode can store shuffle data in remote storage when the disk space is not enough, which could avoid insufficient disk space errors and is only supported when taskmanager.network.hybrid-shuffle.remote.path is configured. The new mode is currently in an experimental phase. It can be set to f [...] - </tr> <tr> <td><h5>taskmanager.network.hybrid-shuffle.memory-decoupling.enabled</h5></td> <td style="word-wrap: break-word;">false</td> <td>Boolean</td> <td>The option is used to make the memory that is used by hybrid shuffle decoupled from the complexity of the job topology and the number of tasks on the task manager. It significantly reduces the chance of the "Insufficient number of network buffers" exception, while the workloads may suffer performance reduction silently.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.num-retained-in-memory-regions-max</h5></td> - <td style="word-wrap: break-word;">1048576</td> - <td>Long</td> - <td>Controls the max number of hybrid retained regions in memory. Note: This option will be ignored if taskmanager.network.hybrid-shuffle.enable-new-mode is set true. </td> - </tr> <tr> <td><h5>taskmanager.network.hybrid-shuffle.remote.path</h5></td> <td style="word-wrap: break-word;">(none)</td> <td>String</td> <td>The option is used to configure the base path of remote storage for hybrid shuffle. The shuffle data will be stored in remote storage when the disk space is not enough. Note: If the option is configured and taskmanager.network.hybrid-shuffle.enable-new-mode is false, this option will be ignored. If the option is not configured and taskmanager.network.hybrid-shuffle.enable-new-mode is true, the remote storage will be disabled.</td> </tr> - <tr> - <td><h5>taskmanager.network.hybrid-shuffle.spill-index-region-group-size</h5></td> - <td style="word-wrap: break-word;">1024</td> - <td>Integer</td> - <td>Controls the region group size(in bytes) of hybrid spilled file data index. Note: This option will be ignored if taskmanager.network.hybrid-shuffle.enable-new-mode is set true.</td> - </tr> <tr> <td><h5>taskmanager.network.max-num-tcp-connections</h5></td> <td style="word-wrap: break-word;">1</td> diff --git a/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java b/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java index 02c048f1445..40c69162731 100644 --- a/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java +++ b/flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java @@ -341,8 +341,14 @@ public class NettyShuffleEnvironmentOptions { // this raw value must be changed correspondingly "taskmanager.memory.framework.off-heap.batch-shuffle.size")); - /** Region group size of hybrid spilled file data index. */ - @Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK) + /** + * Region group size of hybrid spilled file data index. + * + * @deprecated The option is only used in the legacy hybrid shuffle mode. It is deprecated in + * 1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be + * removed in 2.0. + */ + @Deprecated public static final ConfigOption<Integer> HYBRID_SHUFFLE_SPILLED_INDEX_REGION_GROUP_SIZE = key("taskmanager.network.hybrid-shuffle.spill-index-region-group-size") .intType() @@ -355,8 +361,14 @@ public class NettyShuffleEnvironmentOptions { + HYBRID_SHUFFLE_NEW_MODE_OPTION_NAME + " is set true."); - /** Max number of hybrid retained regions in memory. */ - @Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK) + /** + * Max number of hybrid retained regions in memory. + * + * @deprecated The option is only used in the legacy hybrid shuffle mode. It is deprecated in + * 1.20 and will be totally removed in 2.0, as the legacy hybrid shuffle mode will be + * removed in 2.0. + */ + @Deprecated public static final ConfigOption<Long> HYBRID_SHUFFLE_NUM_RETAINED_IN_MEMORY_REGIONS_MAX = key("taskmanager.network.hybrid-shuffle.num-retained-in-memory-regions-max") .longType() @@ -413,9 +425,13 @@ public class NettyShuffleEnvironmentOptions { + "tasks have occupied all the buffers and the downstream tasks are waiting for the exclusive buffers. The timeout breaks" + "the tie by failing the request of exclusive buffers and ask users to increase the number of total buffers."); - /** The option to enable the new mode of hybrid shuffle. */ - @Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK) - @Experimental + /** + * The option to enable the new mode of hybrid shuffle. + * + * @deprecated This option is deprecated in 1.20 and will be totally removed in 2.0, as the + * legacy hybrid shuffle mode will be removed in 2.0. + */ + @Deprecated @Experimental public static final ConfigOption<Boolean> NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE = ConfigOptions.key(HYBRID_SHUFFLE_NEW_MODE_OPTION_NAME) .booleanType()
