WencongLiu commented on code in PR #22975:
URL: https://github.com/apache/flink/pull/22975#discussion_r1270307675
##########
flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java:
##########
@@ -28,6 +28,12 @@
@PublicEvolving
public class NettyShuffleEnvironmentOptions {
+ private static final String HYBRID_SHUFFLE_NEW_MODE_OPTION_NAME =
+ "taskmanager.network.hybrid-shuffle.enable-new-mode";
+
+ private static final String
HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH_OPTION_NAME =
+ "taskmanager.network.hybrid-shuffle.remote.path";
Review Comment:
The two related static configs need to statically refer the name of each
other in the note, so there should be static String names to ensure the random
order.
##########
flink-core/src/main/java/org/apache/flink/configuration/NettyShuffleEnvironmentOptions.java:
##########
@@ -381,6 +393,38 @@ public class NettyShuffleEnvironmentOptions {
+ "tasks have occupied all the buffers and
the downstream tasks are waiting for the exclusive buffers. The timeout breaks"
+ "the tie by failing the request of
exclusive buffers and ask users to increase the number of total buffers.");
+ /** The option to enable the new mode of hybrid shuffle. */
+ @Documentation.Section(Documentation.Sections.ALL_TASK_MANAGER_NETWORK)
+ @Experimental
+ public static final ConfigOption<Boolean>
NETWORK_HYBRID_SHUFFLE_ENABLE_NEW_MODE =
+ ConfigOptions.key(HYBRID_SHUFFLE_NEW_MODE_OPTION_NAME)
+ .booleanType()
+ .defaultValue(true)
+ .withDescription(
+ "The option is used to enable the new mode of
hybrid shuffle, which has resolved existing issues in the legacy mode. First,
the new mode "
+ + "uses less required network memory.
Second, the new mode can store shuffle data in remote storage when the disk
space is not "
+ + "enough, which could avoid insufficient
disk space errors and is only supported when "
+ +
HYBRID_SHUFFLE_REMOTE_STORAGE_BASE_PATH_OPTION_NAME
+ + " is configured. The new mode is
currently in an experimental phase. The option can be set to false to fallback
to the legacy "
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
