Scott Brokaw created YARN-7273:
----------------------------------
Summary: Ability for YARN applications to request lower container
size below yarn.scheduler.minimum-allocation-mb
Key: YARN-7273
URL: https://issues.apache.org/jira/browse/YARN-7273
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Scott Brokaw
Priority: Minor
Currently, a YARN application must request a container size that is between the
yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb
else the request has to be raised by the Application to the
yarn.scheduler.minimum-allocation-mb
yarn.scheduler.minimum-allocation-mb < Requested Container Size >
yarn.scheduler.maximum-allocation-mb
This improvement would either add a new parameter such as
yarn.scheduler.default-allocation-mb or treat
yarn.scheduler.minimum-allocation-mb instead as a global default size if no
container size is requested. This would allow a particular YARN application to
request a smaller amount of resources if it deems it can run with resources
below the minimum allocation dictated by YARN. This allows a significant
amount of resources to be saved as otherwise a YARN application that spawns
many containers can end up consuming many more resources then it needs.
The same idea can be applied to yarn.scheduler.minimum-allocation-vcores.
The argument can be made that you should just set the minimum values lower to
start with, but the idea is that YARN should support a global default value for
applications that don't set it, but allow other applications (if they so
request) to be granted smaller containers. The "problem" quickly compounds the
more containers an application spins up and with memory being a valuable
resource in the cluster, this improvement is looking to maximize the available
resources given an application is "willing and able" to run with a smaller size.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
