Xianyin Xin commented on YARN-3630:

Thanks for your comments, [~vvasudev]!
your patch doesn't check if the calculated interval is greater than the ping 
interval to determine liveliness for the AM and the NM. Is that by design?
It's true we should do that. But in this patch I haven't add the mechanism of 
determining a up limit for the {{nextHeartbeatInterval}}. I think the limit 
should much less than the ping interval which is 10 minutes by default. Other 
hand, do you think a hard configurable limit is accepted? 
With respect to adaptive heartbeats for the NMs - my concern is that the 
proposed solution will lead to behaviour where the NMs will be told to back off 
- the NMs will wait for sometime - the RM will receive a flood of NM updates - 
leading to the NMs being told to back off and so on and so forth. We'll end up 
in a situation where the pings will become clustered around particular time 
intervals, leading to container allocation and release delays. You might be 
better off picking a random interval between the default interval and the 
calculated interval to spread out the NM pings
Thanks for reminding, it's a situation I didn't think much. I think your 
suggestion is a nice choice. 

> YARN should suggest a heartbeat interval for applications
> ---------------------------------------------------------
>                 Key: YARN-3630
>                 URL: https://issues.apache.org/jira/browse/YARN-3630
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.7.0
>            Reporter: Zoltán Zvara
>            Assignee: Xianyin Xin
>            Priority: Minor
>         Attachments: Notes_for_adaptive_heartbeat_policy.pdf, 
> YARN-3630.001.patch.patch, YARN-3630.002.patch
> It seems currently applications - for example Spark - are not adaptive to RM 
> regarding heartbeat intervals. RM should be able to suggest a desired 
> heartbeat interval to applications.

This message was sent by Atlassian JIRA

Reply via email to