[
https://issues.apache.org/jira/browse/FLINK-19343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240704#comment-17240704
]
Jin Xing edited comment on FLINK-19343 at 11/30/20, 11:54 AM:
--------------------------------------------------------------
My team is considering the design of "Remote Shuffle Service", regarding this
"Interactive" FLIP, may I comment by below points ?
# What is the role of ResourceManager ? From my previous understanding, it's
mainly about computing resource management. But I guess "Interactive" will rely
on the ResourceManager#clusterPartitionTracker for shuffle data lifecycle
management. It seems weird. A possible answer is that ResourceManager is a
component whose lifecycle goes across jobs and equals to ApplicationMaster,
thus chosen to manage data across jobs. But I don't see strong causality.
Should we have a separate component to manage lifecycle of shuffle data across
jobs ?
# Additionally current "Interactive" shares shuffle data across jobs by saving
ShuffleDescriptors into table catalog, which bypasses shuffle-service and makes
"Interactive" a pure Table-API level feature. What if we want such interactive
feature on DataStream or DataSet API level in the future? My question is how to
reuse. Should shuffle meta should be managed within the scope of Flink Runtime,
rather than spread out into Sql layer? If we go deeper into this question, the
lifecycle of ShuffleMaster is the same with JobManager in current design, which
makes ShuffleMaster not qualified to manage data sharing across jobs. Related
with my point 1 – should we have a separate component to manage lifecycle of
shuffle data across jobs ?
# In scenario of "Remote Shuffle Service", the lifecycle of TM decouples with
shuffle data. It's not proper to ask the TM#partitionTracker to apply data
release, but should rely on the communication between ShuffleMaster and
ShuffleService.
was (Author: [email protected]):
My team is considering the design of "Remote Shuffle Service", regarding this
"Interactive" FLIP, may I comment by below points ?
1. What is the role of ResourceManager ? From my previous understanding, it's
mainly about computing resource management. But I guess "Interactive" will rely
on the ResourceManager#clusterPartitionTracker for shuffle data lifecycle
management. It seems weird. A possible answer is that ResourceManager is a
component whose lifecycle goes across jobs and equals to ApplicationMaster,
thus chosen to manage data across jobs. But I don't see strong causality.
Should we have a separate component to manage lifecycle of shuffle data across
jobs ?
2. Additionally current "Interactive" shares shuffle data across jobs by saving
ShuffleDescriptors into table catalog, which bypasses shuffle-service and makes
"Interactive" a pure Table-API level feature. What if we want such interactive
feature on DataStream or DataSet API level in the future? My question is how to
reuse. Should shuffle meta should be managed within the scope of Flink Runtime,
rather than spread out into Sql layer? If we go deeper into this question, the
lifecycle of ShuffleMaster is the same with JobManager in current design, which
makes ShuffleMaster not qualified to manage data sharing across jobs. Related
with my point 1 -- should we have a separate component to manage lifecycle of
shuffle data across jobs ?
2. In scenario of "Remote Shuffle Service", the lifecycle of TM decouples with
shuffle data. It's not proper to ask the TM#partitionTracker to apply data
release, but should rely on the communication between ShuffleMaster and
ShuffleService.
> FLIP-36: Support Interactive Programming in Flink
> -------------------------------------------------
>
> Key: FLINK-19343
> URL: https://issues.apache.org/jira/browse/FLINK-19343
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Reporter: Xuannan Su
> Priority: Major
>
> Please refer to the FLIP for any details:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
