[
https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205005#comment-16205005
]
Wangda Tan commented on YARN-6927:
----------------------------------
[~templedf], since MR only supports specifying individual resource types, I
think we can add the support of specifying all types in single field in the
future.
Before this patch, user can specify
{code}
yarn.app.mapreduce.am.resource.mb for AM's memory
yarn.app.mapreduce.am.cpu-vcores for AM's vcores
mapreduce.map/reduce.memory.mb for mapper/reducer's memory
mapreduce.map/reduce.cpu.vcores for mapper/reducer's vcores.
{code}
After this patch, user can specify:
{code}
For AM's memory:
yarn.app.mapreduce.am.resource.mb OR
yarn.app.mapreduce.am.resource.memory-mb = 1024 MB
For AM's vcores:
yarn.app.mapreduce.am.cpu-vcores OR
yarn.app.mapreduce.am.resource.vcores = 5
For Other AM resources:
yarn.app.mapreduce.am.resource.disk-space = 12 GB
For Mapper/Reducer's Memory:
mapreduce.map/reduce.memory.mb OR
mapreduce.map/reduce.resource.memory-mb = 1024 MB
For Mapper/Reducer's vcore:
mapreduce.map/reduce.cpu.vcores OR
mapreduce.map/reduce.resource.vcores = 3
For other Mapper/Reducer's resource:
mapreduce.map/reduce.resource.disk-space = 30 GB
{code}
First,
I think we should throw exception when following options are specified:
{code}
yarn.app.mapreduce.am.resource.memory-mb
yarn.app.mapreduce.am.resource.vcores
mapreduce.map/reduce.resource.memory-mb
mapreduce.map/reduce.resource.vcores
{code}
Because overwriting logic (from the implementation, new options like
mapreduce.map/reduce.resource.vcores overwrite the old one. This is very
confusing to end users.
Second,
It might be better to support specifying space between number and unit. E.g.
1024MB, 1024 MB are all valid and mean the same thing. currently it doesn't
allow adding a space.
Thoughts?
> Add support for individual resource types requests in MapReduce
> ---------------------------------------------------------------
>
> Key: YARN-6927
> URL: https://issues.apache.org/jira/browse/YARN-6927
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Daniel Templeton
> Assignee: Gergo Repas
> Attachments: YARN-6927.000.patch, YARN-6927.001.patch
>
>
> YARN-6504 adds support for resource profiles in MapReduce jobs, but resource
> profiles don't give users much flexibility in their resource requests. To
> satisfy users' needs, MapReduce should also allow users to specify arbitrary
> resource requests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
