[
https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Attachment: MAPREDUCE-3812.patch
Off the failures, only TestRMContainerAllocator looked relevant. Fixed that
test by removing an unnecessary init override (could've also called super.init
but the whole method override was unneeded, I had it carried from the earlier
patch).
Here are new test results, from those that failed, per project:
*YARN Server*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.472 sec
Running
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.826 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.453 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.21 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.997 sec
Results :
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
{code}
*MapReduce Client App*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.106 sec
Results :
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
{code}
*MR JobClient*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.mapred.TestClientRedirect
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.422 sec
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.899 sec
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
{code}
All failed tests have passed locally. On jenkins it looks like some other test
failing to shutdown YARN services in teardown methods is causing this issue.
Resubmitting.
> Lower default allocation sizes, fix allocation configurations and document
> them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt,
> MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt,
> MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch,
> MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like
> MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within
> 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work
> around the issues with AM heap. Now that those are fixed, we should go back
> to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes
> no sense as there's no way to run multiple schedulers anyway. Switch configs
> to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it
> (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be
> left at such values, we should lower the minimum alloc to 128 MB to allow
> special requests of low memory out of the box itself. Shouldn't impact MR App
> in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
