[
https://issues.apache.org/jira/browse/YARN-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291951#comment-16291951
]
Wangda Tan commented on YARN-7457:
----------------------------------
[~kkaranasos], thanks for posting the comment. I agree that missed
opportunistic is not a good protocol. The cost function is an interesting idea,
with that we can even consider crazy ideas like training a deep neural network
to do the placement in the future.
I'm not sure if specifying urgency number is a good idea or not, the time-based
delay is easy for people to understand.
In general, since we're all agree that delay scheduling logic should be
decoupled, I propose to do it as the first step and discuss more while working
on that.
bq. I also see the need for each application to specify how "urgent" its
requests are. That is, one app might want to relax locality after 1 sec and
another might want to relax after 1 min.
Agree. This is why we introduced delayed-or in YARN-6592. In addition to that,
I think we can add a common String to String scheduling hint to
AppSubmissionContext/RegisterAMRequest for legacy apps which using
ResourceRequest and other tasks (such as YARN-7494) can benefit from the
scheduling hint maps.
> Delay scheduling should be an individual policy instead of part of scheduler
> implementation
> -------------------------------------------------------------------------------------------
>
> Key: YARN-7457
> URL: https://issues.apache.org/jira/browse/YARN-7457
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Tao Yang
>
> Currently, different schedulers have slightly different delay scheduling
> implementations. Ideally we should make delay scheduling independent from
> scheduler implementation. Benefits of doing this:
> 1) Applications can choose which delay scheduling policy to use, it could be
> time-based / missed-opportunistic-based or whatever new delay scheduling
> policy supported by the cluster. Now it is global config of scheduler.
> 2) Make scheduler implementations simpler and reusable.
> h2. {color:red}Running design doc:
> https://docs.google.com/document/d/1rY-CJPLbGk3Xj_8sxre61y2YkHJFK8oqKOshro1ZY3A/edit#heading=h.xnzvh9nn283a{color}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
