[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168420#comment-16168420
]
Daniel Templeton commented on YARN-2162:
----------------------------------------
My comments:
# In {{AllocationFileLoaderService.loadQueue()}}, instead of removing the max >
min check, can we make it conditional? Use {{fitsIn()}} if they're both
absolute and numerical comparison if they're both percentages, and ignore the
check otherwise, maybe with a warning. Similar in
{{AllocationConfiguration.getMaxResources()}}. Even though
{{FSQueue.getMaxShare()}} does the check, it would be nice to have the warning
earlier while you still have the configuration details to put in the message.
# Looks like {{AllocationConfiguration.DEFAULT_CALCULATOR}} is unused. Not
your fault, so you can file a newbie JIRA if you prefer.
# In {{AllocationConfiguration.initFSQueue()}}, you should probably change all
the metrics inits to use {{queue.get...()}}.
# In the tests, why do we need to set the calculator on the mock?
# It seems like we should use "vcores" instead of "cpu" for consistency.
# In {{FairSchedulerConfiguration.findPercentage()}}, you can probably drop the
trailing while space in the pattern. You also don't need the group 2
parentheses. And the first '*' can probably be a '+', but you might have done
that intentionally.
# In {{TestFairSchedulerConfiguration.testParseResourceConfigValue()}}, you're
missing some negative tests, like one percentage and one absolute.
# Even though you have the tests in {{testParseResourceConfigValue()}}, it
would probably be a good idea to add some tests for {{ResourceConfiguration}}
itself.
# Have you considered performance impact? Every call to {{getMaxShare()}}
causes a read lock to be locked and unlocked and clones a resource. That
sounds kinda heavy to me for how often it gets called. Probably worth testing.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -----------------------------------------------------------------------------------------
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
> Reporter: Ashwin Shankar
> Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch,
> YARN-2162.003.patch, YARN-2162.004.patch, YARN-2162.005.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
