[
https://issues.apache.org/jira/browse/MAPREDUCE-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749278#action_12749278
]
Jothi Padmanabhan commented on MAPREDUCE-884:
---------------------------------------------
I think I know what is happening here. All the maps are shuffled to memory, but
when the last map is shuffled, it just happens that the memory limit is reached
and a memory to disk merge is triggered and all the output touches the disk. As
a result, spill counter is the same as map output records. A simple fix would
be to increase the number of maps so that the last few maps reside in memory
and so the assert condition is met.
I will fix this as a part of MAPREDUCE-157
> TestReduceFetchFromPartialMem fails sometimes
> ---------------------------------------------
>
> Key: MAPREDUCE-884
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-884
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Reporter: Amar Kamat
>
> TestReduceFetchFromPartialMem failed with the following exception trace :
> {code}
> Expected some records not spilled during reduce40980)
> junit.framework.AssertionFailedError: Expected some records not spilled
> during reduce40980)
> at
> org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:94)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
> at junit.extensions.TestSetup.run(TestSetup.java:27)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
