Sxnan commented on code in PR #19653:
URL: https://github.com/apache/flink/pull/19653#discussion_r879216461
##########
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java:
##########
@@ -84,6 +86,33 @@ CompletableFuture<T> registerPartitionWithProducer(
PartitionDescriptor partitionDescriptor,
ProducerDescriptor producerDescriptor);
+ /**
+ * Returns all the shuffle descriptors for the partitions in the
intermediate data set with the
+ * given id.
+ *
+ * @param intermediateDataSetID The id of the intermediate data set.
+ * @return all the shuffle descriptors for the partitions in the
intermediate data set. Null if
+ * not exist.
+ */
+ default Collection<T> getClusterPartitionShuffleDescriptors(
Review Comment:
Thanks for the clarification.
If I understand correctly, the `ResourceManagerPartitionTracker` currently
only tracks the cluster partition in TaskManager. If we want to unify the
cluster partition management, we have to extend the scope of the
`ResourceManagerPartitionTracker` so that it also tracks the cluster partitions
for remote shuffle service and keeps track of the ShuffleDescriptors of the
cluster partitions. The JobMasterClusterPartitionTracker has to register the
ShuffleDescriptors of cluster partitions to the
ResourceManagerPartitionTracker` when the job is finished. And the
ShuffleMaster of the remote shuffle service should be able to report the
cluster partition lost in the remote shuffle cluster to ResourceManager.
Please let me know if I understand correctly, then I will go ahead and make
the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
