[
https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660919#comment-13660919
]
Hitesh Shah commented on YARN-689:
----------------------------------
I am not sure I understand the headroom references that you gave.
I will try and explain my understanding:
There is no headroom that YARN needs. It runs a container via a shell script
that can in turn launch a shell script ( in case of distributed shell ) or a
jvm ( in case of MR ). To run such a container, it does not require a minimum
set of resources. The shell container could run with say 100 MB or even less,
where as the MR case due to how MR tasks work may need anywhere from 1 GB to 2
GB.
The minimum allocation defined at the scheduler level in RM is actually just
the multiplier. Maybe, calling it minimum is confusing and we could change it
to be slot size or multiplier value?
Lets consider 4 application types:
- App1 needs 1.5 GB sized container .
- App2 needs 2 GB sized containers
- App3 needs 400 MB sized containers.
- App4 needs 1 GB sized containers.
>From the above, the simplest answer will be to set the slot size to 512 MB (
>which is currently set using the minimum allocation config property ). Each
>application has its own set of defaults which can then be translated into
>multiples of slot sizes.
Currently, MR has 3 settings:
- Application Master memory : default is 1.5 GB
- map memory : default is 1 GB
- reduce memory : default is 1 GB
Due to yarn's default configs of 1 GB slot size ( and max container size of 8
GB ), the AM ends up taking 2 GB and the maps/reduces take up 1 GB each. To
make the AM use 1.5 GB containers ( instead of 2 GB ), yarn's minimum
allocation (slot size) could be changed to 512 MB.
Question is whether we need an additional minimum configuration on top of the
already available multiplier setting?
> Add multiplier unit to resourcecapabilities
> -------------------------------------------
>
> Key: YARN-689
> URL: https://issues.apache.org/jira/browse/YARN-689
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, scheduler
> Affects Versions: 2.0.4-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Attachments: YARN-689.patch
>
>
> Currently we overloading the minimum resource value as the actual multiplier
> used by the scheduler.
> Today with a minimum memory set to 1GB, requests for 1.5GB are always
> translated to allocation of 2GB.
> We should decouple the minimum allocation from the multiplier.
> The multiplier should also be exposed to the client via the
> RegisterApplicationMasterResponse
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
