[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16620000#comment-16620000
]
Weiwei Yang commented on YARN-8468:
-----------------------------------
Thanks [~bsteinbach] for the updates. A general comment to your patch is,
please minimize the changes in the patch to keep it neat. Format changes,
switch lines etc should be avoid. Otherwise it is not easy to review because we
can't focus on the changes required for this feature. Detail review comments
below:
1. DefaultAMSProcessor
All the changes in DefaultAMSProcessor seems unnecessary
2. FileScheduler.md
{noformat}
maximum resources a queue can allocate for a single container, expressed in
the form of
{noformat}
a extra space between "expressed" and "in"
3. I think we should use consistent names, but I saw some places have "Max
Container Allocation/maxContainerAllocation" and some other places have "Max
Container Resources/maxContainerResources". A bit confusing.
4. The changes of those imports to {{FSLeafQueue}} seems not necessary, please
do not include changes like to put a import to another line etc. Same comment
applies to classes like {{FSParentQueue}}, {{PlacementConstraintProcessor}},
{{QueueProperties}}, {{TestAllocationFileLoaderService}}, {{TestAppManager}},
{{TestCapacityScheduler}}
5. RMServiceUtils.java, line 331 - 337, changes are not necessary
6. SchedulerUtils.java, except the changes in {{normalizeAndvalidateRequest}}
and {{validateResourceRequest}}, please remove rest of changes
7. Lots of unnecessary changes in {{TestRMServerUtils}}
8. Why it even changes {{TestSchedulerNegotiator}} and {{TestAMLaunchFailure}},
both of them are no longer used right?
Besides, there are still 18 checkstyle issues to fix.
Thanks
> Limit container sizes per queue in FairScheduler
> ------------------------------------------------
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 3.1.0
> Reporter: Antal Bálint Steinbach
> Assignee: Antal Bálint Steinbach
> Priority: Critical
> Attachments: YARN-8468.000.patch, YARN-8468.001.patch,
> YARN-8468.002.patch, YARN-8468.003.patch, YARN-8468.004.patch,
> YARN-8468.005.patch, YARN-8468.006.patch, YARN-8468.007.patch,
> YARN-8468.008.patch, YARN-8468.009.patch, YARN-8468.010.patch,
> YARN-8468.011.patch, YARN-8468.012.patch, YARN-8468.013.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
>
> The goal of this ticket is to allow this value to be set on a per queue basis.
>
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
>
> Suggested solution:
>
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability() in both FSParentQueue and
> FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * write JUnit tests.
> * update the scheduler documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
