[jira] [Updated] (YARN-3922) Introduce adaptive heartbeat between RM and NM
[ https://issues.apache.org/jira/browse/YARN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-3922: - Labels: (was: newbie) Introduce adaptive heartbeat between RM and NM --- Key: YARN-3922 URL: https://issues.apache.org/jira/browse/YARN-3922 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager, resourcemanager Reporter: Xiaodi Ke Currently, the communication between RM and NM are based on pull-based heartbeat protocol. Along with the NM heartbeat, it updates the status of containers (i.e. FINISHED container). This also updates the RM’s view of available resource and triggers scheduling. How frequently the NM sends the heartbeat will impact the task throughput and latency of YARN scheduler. Although the heartbeat interval can be configured in yarn-stie.xml, it will increase the load of RM and bring unnecessary overhead if the interval is configured too short. We propose the adaptive heartbeat between RM and NM to achieve a balance between updating NM’s info promptly and minimizing the overhead of extra heartbeats. With adaptive heartbeat, NM still honors the current heartbeat interval and sends the heartbeat regularly. However, a heartbeat will be triggered as soon as any container status is changed. Also a minimum interval can be configured to prevent NM from sending heartbeat too frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3922) Introduce adaptive heartbeat between RM and NM
[ https://issues.apache.org/jira/browse/YARN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaodi Ke updated YARN-3922: Labels: newbie (was: ) Introduce adaptive heartbeat between RM and NM --- Key: YARN-3922 URL: https://issues.apache.org/jira/browse/YARN-3922 Project: Hadoop YARN Issue Type: New Feature Components: nodemanager, resourcemanager Reporter: Xiaodi Ke Labels: newbie Currently, the communication between RM and NM are based on pull-based heartbeat protocol. Along with the NM heartbeat, it updates the status of containers (i.e. FINISHED container). This also updates the RM’s view of available resource and triggers scheduling. How frequently the NM sends the heartbeat will impact the task throughput and latency of YARN scheduler. Although the heartbeat interval can be configured in yarn-stie.xml, it will increase the load of RM and bring unnecessary overhead if the interval is configured too short. We propose the adaptive heartbeat between RM and NM to achieve a balance between updating NM’s info promptly and minimizing the overhead of extra heartbeats. With adaptive heartbeat, NM still honors the current heartbeat interval and sends the heartbeat regularly. However, a heartbeat will be triggered as soon as any container status is changed. Also a minimum interval can be configured to prevent NM from sending heartbeat too frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
