[
https://issues.apache.org/jira/browse/YARN-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150300#comment-16150300
]
Junping Du commented on YARN-7138:
----------------------------------
Thanks [~leftnoteasy] for response.
bq. Regarding to scheduler of K8S, it works quite different comparing to YARN:
K8S supports customized scheduler or even multiple schedulers concurrently
running so POD can choose to use different schedulers when it is submitted to
the cluster.
Agree that it is different so far, but the way how K8S works now may hint the
direction that YARN could leverage in future - especially for long running
services with extremely scale. Taking federation as an example, we are working
on support for one application to ask resources on multiple clusters now. But
technically, there is no hard restraint for apps to ask resource from one kind
of scheduler which means it potentially can ask resource from multiple clusters
which run different schedulers (at least can mix with FS and CS scheduler). If
that becomes true someday, then how far between YARN and K8S in scheduling (at
least for this scenario) then?
bq. In K8S, there's no official "scheduler" API.
That's not true. K8S website
(https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/)
document how to implement a customized scheduler with several pre-define APIs
and these APIs even get versioned (so far is version 1). For more details, you
can refer:
https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/scheduler_interface.go
bq. So we can say, scheduler of K8S makes decision by invoking POD/binding to
assign POD to node. That's the only contract for customized scheduler, which is
not descriptive enough but very simple to maintain compatibility.
Hmm... I agree that K8S scheduler logic could be simpler than YARN so far but
that doesn't have to mean that maintaining compatibility is also simpler.
Actually, maintaining several basic APIs' compatibility is not that hard for
YARN by following current public API update protocol - just instead of changing
long existing APIs but adding new APIs with mark old one as deprecated.
maintaining allocated() definitely belong to this case.
bq. However since YARN scheduler is too much coupled with other RM components,
so it is almost impossible to keep API stability of scheduler, and beyond API,
scheduler need to handle internal RM events, which we never ensure
compatibility before.
That's unfortunately true. However, ideally, a good design should make YARN
scheduler API to be more independent with implementations of these RM events -
the later should always get updated for new scenarios.
bq. To be frank, I don't see big benefit of making scheduler API stable now,
adding customized scheduler is never recommended by Hadoop community and
vendors. And I can expect declare stability of scheduler API could slow down
innovations in scheduler side.
This is not true. CS and FS is not born for the first day when Hadoop comes.
Even for YARN, FS are added much later than CS. How can we assume there is no
better scheduler than CS/FS for YARN in future? About innovation, I don't
understand how declaring stability on several APIs could slow down things here?
I am more worried about less people could be interested to join contribution of
their ideas as discouraged by non-stop changing APIs.
bq. Had some discussions with Carlo Curino / Subru Krishnan / Konstantinos
Karanasos during Hadoop summit regarding to how to better design scheduler
semantics, I suggest to declare scheduler API stability once we reach there.
Can you share more details on these discussions? better to be document
somewhere for broader audiences. We never reach a place that we can safely
declare something is the best and absolutely stable. Instead, we should ask
ourselves - are we comfortable to claim current APIs are good and relative
stable? If not, keep evolving these APIs to a relatively good and stable status
should be first citizen task for YARN community.
> Fix incompatible API change for YarnScheduler involved by YARN-5521
> -------------------------------------------------------------------
>
> Key: YARN-7138
> URL: https://issues.apache.org/jira/browse/YARN-7138
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
> Reporter: Junping Du
> Priority: Critical
>
> From JACC report for 2.8.2 against 2.7.4, it indicates that we have
> incompatible changes happen in YarnScheduler:
> {noformat}
> hadoop-yarn-server-resourcemanager-2.7.4.jar, YarnScheduler.class
> package org.apache.hadoop.yarn.server.resourcemanager.scheduler
> YarnScheduler.allocate ( ApplicationAttemptId p1, List<ResourceRequest> p2,
> List<ContainerId> p3, List<String> p4, List<String> p5 ) [abstract] :
> Allocation
> {noformat}
> The root cause is YARN-5221. We should change it back or workaround this by
> adding back original API (mark as deprecated if not used any more).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
