[
https://issues.apache.org/jira/browse/FLINK-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883481#comment-16883481
]
Guowei Ma commented on FLINK-13132:
-----------------------------------
[~suez1224] and [~ZhenqiuHuang]. Thanks for your detailed explanations. Your
scenario makes sense to me.
1. Except for the jars, how do you deal with the dependency of the user's jobs
when you move from a cluster to another? For example, the checkpoint data of
jobs and the source and sink of user's jobs. Will your deploy service also move
those dependencies for the users?
2. In the HA scenario, we could use the ability of the SubmittedGraphStore to
resolve the non-deterministic problem. Before generating the new JobGraph,
Dispatcher could check the "SubmittedGraphStore" first then to decide whether
to generate the JobGraph or not. I think [~till.rohrmann] could give more
suggestions.
Thanks.
> Allow ClusterEntrypoints use user main method to generate job graph
> -------------------------------------------------------------------
>
> Key: FLINK-13132
> URL: https://issues.apache.org/jira/browse/FLINK-13132
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.8.0, 1.8.1
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Minor
>
> We are building a service that can transparently deploy a job to different
> cluster management systems, such as Yarn and another internal system. It is
> very cost to download the jar and generate JobGraph in the client side. Thus,
> I want to propose an improvement to make Yarn Entrypoints can be configurable
> to use either FileJobGraphRetriever or ClassPathJobGraphRetriever. It is
> actually a long asking TODO in AbstractionYarnClusterDescriptor in line 834.
> https://github.com/apache/flink/blob/21468e0050dc5f97de5cfe39885e0d3fd648e399/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L834
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
