[
https://issues.apache.org/jira/browse/FLINK-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880502#comment-16880502
]
Zhenqiu Huang commented on FLINK-13132:
---------------------------------------
[~maguowei] [~till.rohrmann]
The reason I want to proposal the change is that we are managing 1000+
production jobs for our customers. As a platform, we need to restart all of the
jobs within 5 minutes according to our SLA, no matter it is a cluster
maintenance or it is a transient infrastructure failure.
Beside this, we are moving to a hybrid cloud architecture. So we need to run
jobs to yarn and another internal cluster management scheduler on top of mesos
and k8 for future. Currently, our customers upload jars to a storage management
layer that cross multiple storage systems (for internal and cloud), so that
they can be secure and efficiently access in both environment. We say cost to
download jar into the service and generate JobGraph in client side is that we
need to start a process for each new job within the service. From our
profiling, it is not scalable enough, so that a large number of instance of the
service are needed for the worst case. But the regular QPS for it is just 10
per second. Thus, we want to further optimize the job submission by push the
job graph generation onto ClusterEntrypoint. By this way, the average job
submission time will be reduced with much less resources.
> Allow ClusterEntrypoints use user main method to generate job graph
> -------------------------------------------------------------------
>
> Key: FLINK-13132
> URL: https://issues.apache.org/jira/browse/FLINK-13132
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Affects Versions: 1.8.0, 1.8.1
> Reporter: Zhenqiu Huang
> Assignee: Zhenqiu Huang
> Priority: Minor
>
> We are building a service that can transparently deploy a job to different
> cluster management systems, such as Yarn and another internal system. It is
> very cost to download the jar and generate JobGraph in the client side. Thus,
> I want to propose an improvement to make Yarn Entrypoints can be configurable
> to use either FileJobGraphRetriever or ClassPathJobGraphRetriever. It is
> actually a long asking TODO in AbstractionYarnClusterDescriptor in line 834.
> https://github.com/apache/flink/blob/21468e0050dc5f97de5cfe39885e0d3fd648e399/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L834
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
