[
https://issues.apache.org/jira/browse/FLINK-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220579#comment-17220579
]
Till Rohrmann commented on FLINK-19761:
---------------------------------------
Are you trying to solve the case where the client does not know what partitions
are available and wants to look it up? How exactly would the client be able to
make sense of the result partition ids? If not, then please explain to me what
use case you believe would be facilitated by having the look up method.
> Add lookup method for registered ShuffleDescriptor in ShuffleMaster
> -------------------------------------------------------------------
>
> Key: FLINK-19761
> URL: https://issues.apache.org/jira/browse/FLINK-19761
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Xuannan Su
> Priority: Major
>
> Currently, the ShuffleMaster can register a partition and get the shuffle
> descriptor. However, it lacks the ability to look up the registered
> ShuffleDescriptors belongs to an IntermediateResult by the
> IntermediateDataSetID.
> Adding the lookup method to the ShuffleMaster can make reusing the cluster
> partition more easily. For example, we don't have to return the
> ShuffleDescriptor to the client just so that the other job can somehow encode
> the ShuffleDescriptor in the JobGraph to consume the cluster partition.
> Instead, we only need to return the IntermediateDatSetID and use it to lookup
> the ShuffleDescriptor by another job.
> By adding the lookup method in ShuffleMaster, if we have an external shuffle
> service and the lifecycle of the IntermediateResult is not bounded to the
> cluster, we can look up the ShuffleDescriptor and reuse the
> IntermediateResult by a job running on another cluster even if the cluster
> that produced the IntermediateResult is shutdown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
