[
https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Description:
After a few performance improvements tracked at MAPREDUCE-3561, like
MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB
vmem. We earlier increased AM slot size from 1 slot to two slots to work around
the issues with AM heap. Now that those are fixed, we should go back to 1GB.
This is just a configuration change.
[P.s.]:
- Currently min/max alloc is set at a per-scheduler config level, which makes
no sense as there's no way to run multiple schedulers anyway. Switch configs to
use a generic RM-config.
- The min/max alloc configs aren't documented and we ought to document it (i.e.
MAPREDUCE-4027)
- 1 GB is perhaps too high for a slot's minimum. While job defaults can be left
at such values, we should lower the minimum alloc to 128 MB to allow special
requests of low memory out of the box itself. Shouldn't impact MR App in any
way.
was:
After a few performance improvements tracked at MAPREDUCE-3561, like
MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB
vmem. We earlier increased AM slot size from 1 slot to two slots to work around
the issues with AM heap. Now that those are fixed, we should go back to 1GB.
This is just a configuration change.
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Summary: Lower default allocation sizes, fix allocation
configurations and document them (was: Change default memory slot sizes to be
1.5GB)
Updated title and description
> Lower default allocation sizes, fix allocation configurations and document
> them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt,
> MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt,
> MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like
> MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within
> 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work
> around the issues with AM heap. Now that those are fixed, we should go back
> to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes
> no sense as there's no way to run multiple schedulers anyway. Switch configs
> to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it
> (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be
> left at such values, we should lower the minimum alloc to 128 MB to allow
> special requests of low memory out of the box itself. Shouldn't impact MR App
> in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
