Wangda Tan commented on YARN-4287:

Thanks for updating, some thinkings regarding to your comments:
bq. This is a behavior change, but I can't think of any good cases where 
someone would prefer the old behavior to the new. Let me know if you can think 
of some.
Agree with you, most of your changes are good, I prefer to enable it to get 
better performance. But I can still think some edge cases, and I'd prefer to 
keep old one to avoid some magic things happen :). Let me explain more:

There're several behavior changes in your patch,
1. rack-delay = min (computed-offswitch-delay, configured-rack-delay)
When large configured-rack-delay specified, it uses old behavior. So this is 
safe to me. And I think what you mentioned before:
bq. I didn't separate them in this version of the patch because I still want to 
be able to specify rack-locality-delay BUT have the computed delay take effect 
when an application is not asking for locality OR is really small.
Makes sense to me, I just feel current way to compute offswitch delay need to 
be improved, I will add an example below.

2. node-delay = min(rack-delay, node-delay).
If a cluster has 40 nodes, user requests 3 containers on node1:
Assume the configured-rack-delay=50, 
rack-delay = min(3 (#requested-container) * 1 (#requested-resource-name)  / 40, 
50) = 0.
node-delay = min(rack-delay, 40) = 0
In above example, no matter how rack-delay specified/computed, if we can keep 
the node-delay to 40, we have better chance to get node-local containers 

3. Don't restore missed-opportunity if rack-local container allocated.
The benefit of this change is obvious - we can get faster rack-local container 
allocation. But I feel this can also affect node-local container allocation (If 
the application asks only a small subset of nodes in a rack), may lead to some 
performance regression for locality I/O sensitive applications.

> Capacity Scheduler: Rack Locality improvement
> ---------------------------------------------
>                 Key: YARN-4287
>                 URL: https://issues.apache.org/jira/browse/YARN-4287
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-4287-v2.patch, YARN-4287-v3.patch, YARN-4287.patch
> YARN-4189 does an excellent job describing the issues with the current delay 
> scheduling algorithms within the capacity scheduler. The design proposal also 
> seems like a good direction.
> This jira proposes a simple interim solution to the key issue we've been 
> experiencing on a regular basis:
>  - rackLocal assignments trickle out due to nodeLocalityDelay. This can have 
> significant impact on things like CombineFileInputFormat which targets very 
> specific nodes in its split calculations.
> I'm not sure when YARN-4189 will become reality so I thought a simple interim 
> patch might make sense. The basic idea is simple: 
> 1) Separate delays for rackLocal, and OffSwitch (today there is only 1)
> 2) When we're getting rackLocal assignments, subsequent rackLocal assignments 
> should not be delayed
> Patch will be uploaded shortly. No big deal if the consensus is to go 
> straight to YARN-4189. 

This message was sent by Atlassian JIRA

Reply via email to