[
https://issues.apache.org/jira/browse/FLINK-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17352943#comment-17352943
]
Zhu Zhu commented on FLINK-16069:
---------------------------------
Thanks for the suggestion and sorry for the late reply! [~trohrmann]
I have some findings of the cause of heartbeat timeout. The cause looks to be
that the serialization to {{submitTask()}} via the future executors overwhelmed
the JVM & machine. When the problem happens, there are lots of (in our case, it
is 96 which equals to the CPU cores of the machines) future executor threads
busy doing the serialization, which makes the main thread hard to get enough
CPU timeslices to handle the RPCs. Besides that, akka threads are not able to
handle the generated {{submitTask}} messages in time. Lots of such large
messaged queued in memory will lead to GC problems which add further loads to
CPU.
I tried decreasing the future executor threadpool size to 4 and the heartbeat
timeout problem is gone.
I think the future executor threadpool size is a bit too large if we expect
these threads to do CPU intensity work. Currently it is hardcoded to be all the
available cores of the JVM (via {{Runtime.getRuntime().availableProcessors()}})
and in many cases it is also all the cores of the machine. Maybe we can add a
configurable max limit to it.
Another option is proposed by [~Thesharing] to optimize the process of
{{submitTask()}} by shipping the {{shuffleDescriptors}} via blob service. This
is possible to relieve the heavy computation to {{submitTask()}} in future
executor threads and can also speed up the task submission. He is working on a
PoC at the moment and will share the result soon.
> Creation of TaskDeploymentDescriptor can block main thread for long time
> ------------------------------------------------------------------------
>
> Key: FLINK-16069
> URL: https://issues.apache.org/jira/browse/FLINK-16069
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Reporter: huweihua
> Priority: Major
> Attachments: FLINK-16069-POC-results
>
>
> The deploy of tasks will take long time when we submit a high parallelism
> job. And Execution#deploy run in mainThread, so it will block JobMaster
> process other akka messages, such as Heartbeat. The creation of
> TaskDeploymentDescriptor take most of time. We can put the creation in future.
> For example, A job [source(8000)->sink(8000)], the total 16000 tasks from
> SCHEDULED to DEPLOYING took more than 1mins. This caused the heartbeat of
> TaskManager timeout and job never success.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
