Sunil G commented on YARN-5889:

Thanks [~eepayne]

Yes. for scheduler, 1ms is also smaller. It was a tradeoff to see the 
performance gain and its impact. With SLS test, i could be see good improvement 
  in allocation speed.

Now to bridge the gap, there are 2 cases
- How to make sure that every allocation gets correct and accurate user-limit 
value given computation happens at 1ms?
- In a lousy cluster, how can we save CPU cycles to prevent too much of 
unnecessary computations?

Yes, an ideal way is as suggested by you.
- Any change in resource (allocation and release of container  etc) for a given 
user could set a state variable. This will set off by the computation thread if 
next cycle falls immediate.
- Its not ideal to ask allocation thread to hold till computation. So by seeing 
this state variable, we might need to compute user-limit in same allocation 

I was looking in second step to see how much impact it can cause if user-limit 
is slightly older. We may over allocate or we may under allocate. I think 
under-allocate scenario is fine as we will allocate more from next milli 
second. However overallocate scenario may be a worry. Still we have 
preemptions/opportunistic ways to handle this.

Ideally we were looking to avoid user-limit computation from same allocation 
thread. So after step 1), we can force the user-allocate thread to push for an 
immediate computation. Still there could some exceptionally rare case where 
user-limit thread is doing computation as per release/allocate demand. But 
another allocation thread (heartbeat) may also go in same time frame. If this 
is fine, I could update my patch to handle this case.


> Improve user-limit calculation in capacity scheduler
> ----------------------------------------------------
>                 Key: YARN-5889
>                 URL: https://issues.apache.org/jira/browse/YARN-5889
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Sunil G
>            Assignee: Sunil G
>         Attachments: YARN-5889.v0.patch, YARN-5889.v1.patch, 
> YARN-5889.v2.patch
> Currently user-limit is computed during every heartbeat allocation cycle with 
> a write lock. To improve performance, this tickets is focussing on moving 
> user-limit calculation out of heartbeat allocation flow.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to