Thesharing commented on code in PR #22674:
URL: https://github.com/apache/flink/pull/22674#discussion_r1209723546
##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -104,4 +122,14 @@ public void
markPartitionFinished(IntermediateResultPartition resultPartition) {
checkNotNull(
resultPartitionIdToIndex.get(resultPartition.getPartitionId()))));
}
+
+ private boolean isForceOffload(ShuffleDescriptorAndIndex[]
shuffleDescriptorsToSerialize) {
+ // The unknown shuffle descriptor will be compressed, so don't take it
into account.
+ long numKnownShuffleDescriptors =
Review Comment:
Could we add some tests for the new logics, particularly for this part?
##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -104,4 +122,14 @@ public void
markPartitionFinished(IntermediateResultPartition resultPartition) {
checkNotNull(
resultPartitionIdToIndex.get(resultPartition.getPartitionId()))));
}
+
+ private boolean isForceOffload(ShuffleDescriptorAndIndex[]
shuffleDescriptorsToSerialize) {
Review Comment:
How about `shouldOffload` instead of `isForceOffload`? I think we're not
forced to do this optimization.🤣
##########
flink-runtime/src/main/java/org/apache/flink/runtime/deployment/CachedShuffleDescriptors.java:
##########
@@ -38,6 +39,14 @@
/** {@link ShuffleDescriptor}s cache for a {@link ConsumedPartitionGroup}. */
public class CachedShuffleDescriptors {
+ /**
+ * The threshold to force enable offload shuffle descriptors via blob
server. This is a fixed
+ * value since it is difficult for users to configure. This default value
means JobManager need
+ * to serialized and transport 10000 shuffle descriptors(almost 200KB) to
10000 consumer(2GB in
+ * total)
+ */
+ private static final int FORCE_OFFLOAD_SHUFFLE_DESCRIPTORS_THRESHOLD =
10000 * 10000;
Review Comment:
How about making this option configurable but not exposed to the user manual?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
