[
https://issues.apache.org/jira/browse/YARN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626960#comment-14626960
]
Xiaodi Ke commented on YARN-3922:
---------------------------------
Hi [~vvasudev], thank you for pointing out YARN-3630. YARN-3630 and this ticket
are targeting at different problems.YARN-3630 is trying to slow down the
heartbeat in a large cluster in order not to overwhelm the RM, while our angle
is trying to inform RM about latest info as soon as possible in order to get
better task throughput and lower latency. These two features doesn't conflict
and can work together.
> Introduce adaptive heartbeat between RM and NM
> -----------------------------------------------
>
> Key: YARN-3922
> URL: https://issues.apache.org/jira/browse/YARN-3922
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, resourcemanager
> Reporter: Xiaodi Ke
>
> Currently, the communication between RM and NM are based on pull-based
> heartbeat protocol. Along with the NM heartbeat, it updates the status of
> containers (i.e. FINISHED container). This also updates the RM’s view of
> available resource and triggers scheduling. How frequently the NM sends the
> heartbeat will impact the task throughput and latency of YARN scheduler.
> Although the heartbeat interval can be configured in yarn-stie.xml, it will
> increase the load of RM and bring unnecessary overhead if the interval is
> configured too short.
> We propose the adaptive heartbeat between RM and NM to achieve a balance
> between updating NM’s info promptly and minimizing the overhead of extra
> heartbeats. With adaptive heartbeat, NM still honors the current heartbeat
> interval and sends the heartbeat regularly. However, a heartbeat will be
> triggered as soon as any container status is changed. Also a minimum
> interval can be configured to prevent NM from sending heartbeat too
> frequently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
