[
https://issues.apache.org/jira/browse/YUNIKORN-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260177#comment-17260177
]
Tao Yang commented on YUNIKORN-1:
---------------------------------
Hi, [~wangda] [~wwei] [~wilfreds]
As the refactor for core scheduling process has been done, I would like to
keep this ticket moving. This time I think we can establish a pluggable
mechanism with flexible interfaces at first.
Currently the key steps of scheduling process such as applications/requests
sorting are implemented with fixed process and policies, users may have some
different requirements on them but it's not flexible enough. Moreover, some
steps are relatively time-consuming in every scheduling cycle, we can improve
the performance a lot via different implementations if they are pluggable.
I have tried to improve following places:
(1) requests inside the application which are managed via a map structure.
To make it pluggable, at first we should add a Request interface out of
scheduler/objects package for AllocationAsk to avoid circular dependencies,
then add Requests interface with some common methods like
AddRequest/RemoveRequest/GetRequest in interfaces package, and add a default
implemetation DefaultRequests with map structure for it, at last add
RequestsPlugin interface and its default implementation in plugins package
which helps to generate new requests instance when creating an application.
(2) key steps about sorting applications and requests inside a queue.
Currently sorting applications and sorting requests are separated so that some
requirements can't be satisfied, for example we have 3 apps APP1/APP2, every
application have different priorities(APP1: 3/5, APP2: 2/4), the request
priority is utmost important so that the required sequence may be
APP1/priority=5 > APP2/priority=4 > APP1/priority=3 > APP2/priority=2. Thus I
propose to add QueueRequestManager interface to handle these steps together to
make it more flexible, it may have simple method like SortForAllocation() whose
return is an iterator which can output an application and its requests,
AppRequestIterator/RequestIterator interfaces are added for iterating
applications and requests, QueueRequestManagerPlugin interface is imported as
RequestsPlugin which helps to generate QueueRequestManager instance when
initilizing a queue.
Please refer to the [draft
PR|https://github.com/apache/incubator-yunikorn-core/pull/237] for details.
Hope to hear your thoughts, Thanks.
> Support app/task priority aware scheduling
> ------------------------------------------
>
> Key: YUNIKORN-1
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - scheduler
> Reporter: Weiwei Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
> Attachments: Support priority-based scheduling among applications
> v1.pdf, Support priority-based scheduling among applications v2.pdf, Support
> priority-based scheduling among applications v3.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The scheduler needs to understand app/task priority and make proper decisions
> while allocating resources. I think we don't need to introduce a very complex
> model when defining priorities, e.g supporting both queues, app, and task
> level priorities, it will be hard for users to consume. Instead, we need to
> find a clean, straightforward approach to handle priorities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
