[
https://issues.apache.org/jira/browse/HADOOP-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720392#action_12720392
]
Chris Douglas commented on HADOOP-6029:
---------------------------------------
bq. In ReduceTask.ReduceCopier.ShuffleRamManager, a memory reservation is done
for in-memory shuffle, and that uses Runtime.getRuntime().maxMemory(). The
return value of this seems to be machine-dependent.
This should be rigged by TestReduceFetch in mapred.child.java.opts, and match
{{-Xmx128m}}.
{{testReduceFromPartialMem}} is an awkward test to write. Its intent is to
configure the reduce so that- presented with a set of crafted map outputs- it
will make a particular guess about how to optimize its I/O. If we can't rig the
total memory because \-Xmx has a machine-dependent interpretation, then writing
such a test will be a real pain with our current set of configuration options.
We could add a parameter for the memory reservation that defaults to querying
the runtime; that would let us be certain of our memory limit, but not burden
the user with setting it. I'm really surprised that setting \-Xmx doesn't work,
though...
The failure for {{testReduceFromMem}} suggests this should use the counters
added in HADOOP-2774 rather than the FileSystem counters to validate the result.
> TestReduceFetch failed.
> -----------------------
>
> Key: HADOOP-6029
> URL: https://issues.apache.org/jira/browse/HADOOP-6029
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Tsz Wo (Nicholas), SZE
> Attachments:
> FAILING-PARTIALMEM-TEST-org.apache.hadoop.mapred.TestReduceFetch.txt,
> TEST-org.apache.hadoop.mapred.TestReduceFetch.txt
>
>
> {noformat}
> Testcase: testReduceFromMem took 23.625 sec
> FAILED
> Non-zero read from local: 83
> junit.framework.AssertionFailedError: Non-zero read from local: 83
> at
> org.apache.hadoop.mapred.TestReduceFetch.testReduceFromMem(TestReduceFetch.java:289)
> at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
> at junit.extensions.TestSetup.run(TestSetup.java:27)
> {noformat}
> Ran TestReduceFetch a few times on a clean trunk. It failed consistently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
