zhijiangW commented on a change in pull request #8789: [FLINK-12890] Add
partition lifecycle related Shuffle API
URL: https://github.com/apache/flink/pull/8789#discussion_r295767564
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##########
@@ -41,4 +44,17 @@
CompletableFuture<T> registerPartitionWithProducer(
PartitionDescriptor partitionDescriptor,
ProducerDescriptor producerDescriptor);
+
+ /**
+ * Release any external resources occupied by the given partition.
+ *
+ * <p>This call triggers release of any resources which are occupied by
the given partition in the external systems
+ * outside of the producer executor. This is mostly relevant for the
batch jobs and blocking result partitions
+ * for which {@link
ResultPartitionDeploymentDescriptor#isReleasedOnConsumption()} returns {@code
false}.
+ * The producer local resources are managed by {@link
ShuffleDescriptor#hasLocalResources()} and
+ * {@link ShuffleEnvironment#releasePartitions(Collection)}.
+ *
+ * @param shuffleDescriptor shuffle descriptor of the result partition
to release externally.
+ */
+ void releasePartitionExternally(T shuffleDescriptor);
Review comment:
I am a bit confused about the keyword `externally`.
IMO, we provide different ways for releasing partitions. One is based on
consumption once which already exists. The other is based on JM's decision when
to release a partition in batch job/blocking partition, so the `ShuffleMater`
provides an interface for JM calling, then `ShuffleMater` communicates with
relevant `ShuffleEnvironment` which already provides `releasePartition` to do
so.
No matter which way we provide, the target is to release partition, the
resources would be naturally freed as a result of partition release. For
blocking partition case, it might also free some local resources in
TaskExecutor, so I think we do not need to distinguish the external/internal
resources here. Might only call `ShuffleMaster#releasePartition` and
`ShuffleEnvironment#releasePartition` separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
