azagrebin commented on a change in pull request #8789: [FLINK-12890] Add
partition lifecycle related Shuffle API
URL: https://github.com/apache/flink/pull/8789#discussion_r296643481
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/ShuffleMaster.java
##########
@@ -41,4 +44,17 @@
CompletableFuture<T> registerPartitionWithProducer(
PartitionDescriptor partitionDescriptor,
ProducerDescriptor producerDescriptor);
+
+ /**
+ * Release any external resources occupied by the given partition.
+ *
+ * <p>This call triggers release of any resources which are occupied by
the given partition in the external systems
+ * outside of the producer executor. This is mostly relevant for the
batch jobs and blocking result partitions
+ * for which {@link
ResultPartitionDeploymentDescriptor#isReleasedOnConsumption()} returns {@code
false}.
+ * The producer local resources are managed by {@link
ShuffleDescriptor#hasLocalResources()} and
+ * {@link ShuffleEnvironment#releasePartitions(Collection)}.
+ *
+ * @param shuffleDescriptor shuffle descriptor of the result partition
to release externally.
+ */
+ void releasePartitionExternally(T shuffleDescriptor);
Review comment:
I agree the problem is not easy and the resulting tables in our discussion
should really help the understanding :)
We could also define that the shuffle user (JM/TM) has to guarantee to
always call both methods `ShuffleMaster#releasePartitions` and
`ShuffleEnvironment#releasePartitions` then an implementer could decide which
cleanup to do in which method but this is again not true because once the TM is
gone, `ShuffleEnvironment#releasePartitions` cannot be called.
`ShuffleMaster` can already communicate directly to `ShuffleEnvironment`. It
just does not have to for only `ShuffleEnvironment#releasePartitionsLocally`
and the naming states it clearly. If `SD#storesLocalResourcesOn` then the
shuffle user (JM/TM) guarantees doing the RPC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
