Antal Bálint Steinbach created YARN-8468:
--------------------------------------------
Summary: Limit container sizes per queue in FairScheduler
Key: YARN-8468
URL: https://issues.apache.org/jira/browse/YARN-8468
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 3.1.0
Reporter: Antal Bálint Steinbach
Assignee: Antal Bálint Steinbach
When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb" to
limit the overall size of a container. This applies globally to all containers
and cannot be limited by queue or and is not scheduler dependent.
The goal of this ticket is to allow this value to be set on a per queue basis.
The use case: User has two pools, one for ad hoc jobs and one for enterprise
apps. User wants to limit ad hoc jobs to small containers but allow enterprise
apps to request as many resources as needed. Setting
yarn.scheduler.maximum-allocation-mb sets a default value for maximum container
size for all queues and setting maximum resources per queue with
“maxContainerResources” queue config value.
Suggested solution:
All the infrastructure is already in the code. We need to do the following:
* add the setting to the queue properties for all queue types (parent and
leaf), this will cover dynamically created queues.
* if we set it on the root we override the scheduler setting and we should not
allow that.
* make sure that queue resource cap can not be larger than scheduler max
resource cap in the config.
* implement getMaximumResourceCapability(String queueName) in the FairScheduler
* implement getMaximumResourceCapability() in both FSParentQueue and
FSLeafQueue as follows
* expose the setting in the queue information in the RM web UI.
* expose the setting in the metrics etc for the queue.
* write JUnit tests.
* update the scheduler documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
