zentol commented on a change in pull request #9960: [FLINK-14476] Extend
PartitionTracker to support promotions
URL: https://github.com/apache/flink/pull/9960#discussion_r340673375
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PartitionTrackerImpl.java
##########
@@ -147,22 +160,35 @@ public boolean isPartitionTracked(final
ResultPartitionID resultPartitionID) {
return Optional.of(partitionInfo);
}
+ private Stream<PartitionInfo>
stopTrackingPartitionsAndRetrievePartitionInfo(Collection<ResultPartitionID>
resultPartitionIds) {
+ return resultPartitionIds.stream()
+ .map(this::internalStopTrackingPartition)
+ .flatMap(PartitionTrackerImpl::asStream);
+ }
+
private void internalReleasePartitions(
ResourceID potentialPartitionLocation,
Collection<ResultPartitionDeploymentDescriptor>
partitionDeploymentDescriptors) {
internalReleasePartitionsOnTaskExecutor(potentialPartitionLocation,
partitionDeploymentDescriptors);
-
internalReleasePartitionsOnShuffleMaster(partitionDeploymentDescriptors);
+
internalReleasePartitionsOnShuffleMaster(partitionDeploymentDescriptors.stream());
+ }
+
+ private void internalReleaseOrPromotePartitions(
+ ResourceID potentialPartitionLocation,
+ Collection<ResultPartitionDeploymentDescriptor>
partitionDeploymentDescriptors) {
+
+
internalReleaseOrPromotePartitionsOnTaskExecutor(potentialPartitionLocation,
partitionDeploymentDescriptors);
+
internalReleasePartitionsOnShuffleMaster(excludePersistentPartitions(partitionDeploymentDescriptors));
Review comment:
not necessarily.
My current understanding is that we distinguish between internal and
external shuffle services.
Internal shuffle services work like netty, in that cluster partitions bound
to the life-cycle of the cluster. We need dedicated promotion to ensure the TE
doesn't clean things up when the JM exits, but this is handled outside the
shuffle service.
External shuffle services effectively already behave as if all partitions
are cluster partitions, as they do not have a cleanup routine in case of a JM
shutdown like the TaskExecutors do.
Thus, for external shuffle services we don't need promote calls; they are
cluster partitions already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
