[
https://issues.apache.org/jira/browse/YARN-4902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410294#comment-15410294
]
Wangda Tan commented on YARN-4902:
----------------------------------
[~kkaranasos],
bq. LRA planning is much more than an implementation. Think of it as planning
multiple applications at once. This is something that the scheduler cannot do,
no matter what its implementation is...
I know the implementation of YARN-1051, I participated reviewing JIRAs of
YARN-1051 from the beginning.
I would say we should have a unified API for client to use instead of continue
adding new APIs. Breaking down APIs to different sets is especially bad for new
feature incubation, this is one of the major reason we created YARN-4902.
Planning more than one applications at once should be an enhancement inside
scheduler instead of something visible by end user.
And I'm not sure if we should create a new LRA planner for that, if you're
thinking to implement it using approach of YARN-1051, you may need to reserve a
chunk of resources for "LRA scheduling". This approach may add additional
latency to scheduling (since heavier computation) and reduce throughput (since
additional resources need to be reserved). But this approach looks more
reasonable while doing YARN-1051 -- it is straightforward reserve some
resources for reservation system.
I also have mentioned partial-global-scheduling vs. global-optimal-scheduling
in design doc of YARN-5139:
https://issues.apache.org/jira/secure/attachment/12822180/YARN-5139-Global-Schedulingd-esign-and-implementation-notes.pdf.
You can take a look at it if you're interested.
I can understand your proposal may look different from my guess above, we can
discuss more once you have a more concrete design for that.
bq. I would say that it is the other way around..
I'm not care too much about if we should support cardinality via GUTS API or
support anti-affinity via cardinality syntaxes. We should choose a more
generic/extensible API which can support both.
And I just created a branch YARN-4902 and created sub task YARN-5478, we can
discuss more about Java API definition in that JIRA.
> [Umbrella] Generalized and unified scheduling-strategies in YARN
> ----------------------------------------------------------------
>
> Key: YARN-4902
> URL: https://issues.apache.org/jira/browse/YARN-4902
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Wangda Tan
> Attachments: Generalized and unified scheduling-strategies in YARN
> -v0.pdf, LRA-scheduling-design.v0.pdf, YARN-5468.prototype.patch
>
>
> Apache Hadoop YARN's ResourceRequest mechanism is the core part of the YARN's
> scheduling API for applications to use. The ResourceRequest mechanism is a
> powerful API for applications (specifically ApplicationMasters) to indicate
> to YARN what size of containers are needed, and where in the cluster etc.
> However a host of new feature requirements are making the API increasingly
> more and more complex and difficult to understand by users and making it very
> complicated to implement within the code-base.
> This JIRA aims to generalize and unify all such scheduling-strategies in YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
