Craig Welch commented on YARN-3318:

bq. ...Do we really see non-comparator based ordering-policy. We are 
unnecessarily adding two abstractions - adding policies and comparators...

In the context of the code so far, the "comparator based" approach is specific 
to compounding comparators to achieve functionality (priority + fifo, fair + 
fifo, etc).  This was the initial motivation for the two level api & 
configuration, the broader surface of the policy which would allow for 
different collection types, sorting on demand, etc, (the original policy) and 
the narrower one within that (comparator) for the cases where comparator logic 
was sufficient, e.g. sharing a collection (for composition) and a collection 
type (a tree, for efficient resorting of individual elements when required) was 

The two level api & configuration was not well received.  Offline Wangda has 
indicated that he thinks there are policies coming up which will need the 
wider, initial api, with control over the collection, sorting, etc.  Supporting 
policy composition for those cases would be very awkward & is not really worth 

The various competing tradeoffs, the aversion to a multilevel api, the need for 
the higher level api, and the ability to compose policies creates something of 
a tension, I don't think it's realistic to try and accomplish it all together, 
the result will be Frankensteinian at best.  Something has to go.  Originally, 
I chose the multilevel api to resolve the dilemma, I like that choice, it seems 
unpopular with the crowd.  Given that, the other "optional" dynamic is the 
ability to compose policies (there's no requirement for either of these as far 
as I can tell, it is a "bonus feature").  While I like the composition 
approach, it can't be maintained as such with the broader api and without the 
multilevel config/api.  As one of these has to go and it appears it can't be 
the broader api or the multilevel api I suppose it will have to be composition. 
 Internally there can be some composition of course, but it won't be 
transparent/exposed/configurable as it was initially.

I'll put out a patch with that in a bit.

> Create Initial OrderingPolicy Framework and FifoOrderingPolicy
> --------------------------------------------------------------
>                 Key: YARN-3318
>                 URL: https://issues.apache.org/jira/browse/YARN-3318
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>            Reporter: Craig Welch
>            Assignee: Craig Welch
>         Attachments: YARN-3318.13.patch, YARN-3318.14.patch, 
> YARN-3318.17.patch, YARN-3318.34.patch, YARN-3318.35.patch, 
> YARN-3318.36.patch, YARN-3318.39.patch, YARN-3318.45.patch, 
> YARN-3318.47.patch, YARN-3318.48.patch
> Create the initial framework required for using OrderingPolicies and an 
> initial FifoOrderingPolicy

This message was sent by Atlassian JIRA

Reply via email to