Karthik Kambatla commented on YARN-3630:

Number of events in the dispatcher is definitely a good indicator of how busy 
the scheduler is. We could use it as one of the factors. Other factors:
# Some apps are more tolerant of longer intervals than others and we should 
have a provision to specify an upper-limit on the interval. 
# Weight/capacity of the queue based on whether it is FairScheduler/ 
# How starved an application is. In FairScheduler, that translates to 
min(fairshare - current-allocation, pending-resources).

Other things we need to consider:
# How do we plan to enforce the scheduler doesn't hear from applications sooner 
than the specified interval? The AM can always choose to ignore, right? 
# Are we considering automatically slowing down the NM heartbeats as well? With 
continuous/asynchronous scheduling enabled, I suppose slowing down NM 
heartbeats could be better than slowing down AM heartbeats. We should be 
careful here though - we need to take into account how used a node is. If a 
node is more allocated, slowing heartbeats could lead to delay in noticing 
completed containers. 

> YARN should suggest a heartbeat interval for applications
> ---------------------------------------------------------
>                 Key: YARN-3630
>                 URL: https://issues.apache.org/jira/browse/YARN-3630
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager, scheduler
>    Affects Versions: 2.7.0
>            Reporter: Zoltán Zvara
>            Assignee: Xianyin Xin
>            Priority: Minor
> It seems currently applications - for example Spark - are not adaptive to RM 
> regarding heartbeat intervals. RM should be able to suggest a desired 
> heartbeat interval to applications.

This message was sent by Atlassian JIRA

Reply via email to