[
https://issues.apache.org/jira/browse/FLINK-23905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402729#comment-17402729
]
huntercc edited comment on FLINK-23905 at 8/23/21, 9:53 AM:
------------------------------------------------------------
Thanks for your reply and practical advice, [~trohrmann]. In fact, we have
adopted a similar method by configuring the yarn.ship-files parameter, which
greatly shortens the time spent in this step. I'm worried that there will be
dependency conflicts in this way, especially when we use the yarn session mode.
I venture to suppose that it would be better if this part of the work could be
transparent to users.
was (Author: huntercc):
Thanks your reply and practical advice, [~trohrmann]. In fact, we have adopted
a similar method by configuring the yarn.ship-files parameter, which greatly
shortens the time spent in this step. I'm worried that there will be dependency
conflicts in this way, especially when we use the yarn session mode. I venture
to suppose that it would be better if this part of the work could be
transparent to users.
> Reduce the load on JobManager when submitting large-scale job with a big user
> jar
> ---------------------------------------------------------------------------------
>
> Key: FLINK-23905
> URL: https://issues.apache.org/jira/browse/FLINK-23905
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: huntercc
> Priority: Major
>
> As described in FLINK-20612 and FLINK-21731, there are some time-consuming
> steps in the job startup phase. Recently, we found that when submitting a
> large-scale job with a large user jar, the time spent on changing the status
> of a task from deploying to running accounts for a high proportion of the
> total time-consuming.
> In the task initialization stage, the user jar needs to be pulled from the
> JobManager through BlobService. JobManager has to allocate a lot of computing
> power to distribute the files, which leads to a heavy load in the start-up
> stage. More generally, JobManager fails to respond to the RPC request sent by
> the TaskManager side in time due to high load, causing some timeout
> exceptions, such as akka timeout exception, which leads to job restart and
> further prolongs the start-up time of the job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
