zhuzhurk commented on a change in pull request #18376:
URL: https://github.com/apache/flink/pull/18376#discussion_r790282508
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SsgNetworkMemoryCalculationUtils.java
##########
@@ -148,6 +161,78 @@ private static TaskInputsOutputsDescriptor
buildTaskInputsOutputsDescriptor(
return ret;
}
+ private static Map<IntermediateDataSetID, Integer>
getMaxInputChannelNumsForDynamicGraph(
+ ExecutionJobVertex ejv) {
+
+ Map<IntermediateDataSetID, Integer> ret = new HashMap<>();
+ for (IntermediateResult consumedResult : ejv.getInputs()) {
+ ret.put(consumedResult.getId(),
getMaxInputChannelNumForResult(ejv, consumedResult));
+ }
+
+ return ret;
+ }
+
+ private static Map<IntermediateDataSetID, Integer>
getMaxSubpartitionNumsForDynamicGraph(
+ ExecutionJobVertex ejv) {
+
+ Map<IntermediateDataSetID, Integer> ret = new HashMap<>();
+
+ for (IntermediateResult intermediateResult :
ejv.getProducedDataSets()) {
+ final int maxNum =
+ Arrays.stream(intermediateResult.getPartitions())
+
.map(IntermediateResultPartition::getNumberOfSubpartitions)
+ .reduce(0, Integer::max);
+ ret.put(intermediateResult.getId(), maxNum);
+ }
+
+ return ret;
+ }
+
+ @VisibleForTesting
+ static int getMaxInputChannelNumForResult(
+ ExecutionJobVertex ejv, IntermediateResult consumedResult) {
+ DistributionPattern distributionPattern =
consumedResult.getConsumingDistributionPattern();
+
+ if (distributionPattern == DistributionPattern.ALL_TO_ALL) {
+ int numChannelsToConsumePerPartition =
+
getMaxNumOfChannelsForConsuming(consumedResult.getPartitions()[0]);
+ int numConsumedPartitions =
consumedResult.getNumberOfAssignedPartitions();
+ return numChannelsToConsumePerPartition * numConsumedPartitions;
+
+ } else if (distributionPattern == DistributionPattern.POINTWISE) {
+ int numPartitions = consumedResult.getNumberOfAssignedPartitions();
+ int numConsumers = ejv.getParallelism();
+ // when using dynamic graph, all partitions have the same number
of subpartitions
Review comment:
Another thing I'm thinking is that, can we reuse the logic of
`getMaxInputChannelNums`? Because the edge connections should be the same no
matter if it's a dynamic graph or not. The only difference is the subpartition
range that one edge consumes. Furthermore, execution edges from the same
`IntermediateResult` to the same sub-task consumes the same range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
