[
https://issues.apache.org/jira/browse/YARN-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790879#comment-13790879
]
Sandy Ryza commented on YARN-1241:
----------------------------------
Thanks for the review Karthik. Will post a rebased patch on latest trunk with
your suggestions except where discussed below.
bq. Should we make this a synchronized method along with other accessors of
runnable and non-runnable appScheds? In case, we make the scheduler
multi-threaded?
I think you're right that this will ultimately need to be synchronized when we
remove the super-coarse synchronization that we have currently. But, as a bunch
of methods in similar situations currently aren't synchronized, I'd rather do
it in a separate JIRA where we can take this on all at once.
bq. Making a mental note. The assumption here is that we don't use demand to
determine fairness between schedulables. Correct?
Correct
bq. Should we even consider the demand for non-runnable schedulables?
My thinking I think it's more useful for a cluster operator to know about the
total amount of resources demanded by all jobs submitted to the cluster. E.g.
someone might look at a queue's demand because they want to know whether
anybody in it is waiting for anything, and leaving out the non-runnable apps
would mask this from them.
bq. MaxRunningAppsEnforcer should be singleton?
As there's only a single FairScheduler per RM, there will only end up being
single instance. If for some reason we wanted to run multiple FairSchedulers in
a process (for testing maybe), we would need a MaxRunningAppsEnforcer for each.
bq. I am assuming the ListMultimap choice is to maintain the ordering and give
preference to the earliest submitted job within a queue. Wondering if it would
make sense to use any other metric than submit time for "improved fairness"?
Right. Definitely sounds conceivable to me that we might want to order them by
something else in the future, but I don't we need to make the data structure
more general until then.
bq. IIUC, we are trying to check if the completing of this app makes any other
applications runnable now? Shouldn't we break out of the loop as soon as we
encounter a queue with zero non-runnable apps?
Yeah that's what we are trying to check. childqueueX might have no pending
apps itself, but if a queue higher up in the hierarchy parentqueueY has a
maxRunningApps set, an app completion in childqueueX could allow an app in some
other distant child of parentqueueY to become runnable. I'll update the
comment to try to explain a little more of what's being done.
bq. Also, would be nice to have separate unit tests for MaxRunningAppsEnforcer.
You're right. I will add a couple in.
> In Fair Scheduler maxRunningApps does not work for non-leaf queues
> ------------------------------------------------------------------
>
> Key: YARN-1241
> URL: https://issues.apache.org/jira/browse/YARN-1241
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.1.0-beta
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Attachments: YARN-1241-1.patch, YARN-1241-2.patch, YARN-1241-3.patch,
> YARN-1241.patch
>
>
> Setting the maxRunningApps property on a parent queue should make it that the
> sum of apps in all subqueues can't exceed it
--
This message was sent by Atlassian JIRA
(v6.1#6144)
