[
https://issues.apache.org/jira/browse/FLINK-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241382#comment-17241382
]
Xuannan Su commented on FLINK-19343:
------------------------------------
Hi, Jin Xing. Thanks for your comments.
1. I agree with you that it is indeed weird for the "Interactive" relies on the
clusterPartitionTracker for managing the lifecycle of the cluster partition,
which is also a form of shuffle data. And to support cache across jobs, we need
to have a component whose lifecycle outlive the job to manage the shuffle data.
2. TBH, also, I wouldn't say I like the idea to pass the ShuffleDescriptor back
to the client-side. But at that time being, as you say, we do not have a
separate component to manage the lifecycle of shuffle data across jobs in the
runtime. Therefore, the decision is made to support share shuffle data across
jobs. To support caching in DataStream, it is a more clear design to have a
component at runtime scope to manage the shuffle data across jobs.
3. I don't think the remote shuffle data should be managed by the
PartitionTracker as well. Instead, I think the ClusterPartition is just a kind
of shuffle data and therefore should be managed by the ShuffleService.
I am pulling in [~chesnay]. He may have more insight from the ClusterPartition
perspective.
> FLIP-36: Support Interactive Programming in Flink
> -------------------------------------------------
>
> Key: FLINK-19343
> URL: https://issues.apache.org/jira/browse/FLINK-19343
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Reporter: Xuannan Su
> Priority: Major
>
> Please refer to the FLIP for any details:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
