[
https://issues.apache.org/jira/browse/MAPREDUCE-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803127#action_12803127
]
Aaron Kimball commented on MAPREDUCE-1172:
------------------------------------------
I just took the patch from MAPREDUCE-433 and merged it so that it would apply
to my 20-branch. The testReduceFromPartialMem testcase still fails. I'm seeing
the condition {{spill == 2 * out}}, which is violating the assertion {{spill <
2 * out}}.
After merging the patch in, this is what my testcase method looks like:
{code}
public void testReduceFromPartialMem() throws Exception {
final int MAP_TASKS = 5;
JobConf job = mrCluster.createJobConf();
job.setNumMapTasks(MAP_TASKS);
job.setInt("mapred.inmem.merge.threshold", 0);
job.set("mapred.job.reduce.input.buffer.percent", "1.0");
job.setInt("mapred.reduce.parallel.copies", 1);
job.setInt("io.sort.mb", 10);
job.set("mapred.child.java.opts", "-Xmx128m");
job.setInt("mapred.job.reduce.total.mem.bytes", 128 << 20);
job.set("mapred.job.shuffle.input.buffer.percent", "0.14");
job.setNumTasksToExecutePerJvm(1);
job.set("mapred.job.shuffle.merge.percent", "1.0");
Counters c = runJob(job);
final long out =
c.findCounter(Task.Counter.MAP_OUTPUT_RECORDS).getCounter();
final long spill = c.findCounter(Task.Counter.SPILLED_RECORDS).getCounter();
final long twiceOut = 2 * out;
assertTrue("Expected some records not spilled during reduce (spill="
+ spill + ", 2*out=" + twiceOut + ")",
spill < twiceOut); // spilled map records, some records at the reduce
}
{code}
> TestReduceFetch failed in 0.20
> ------------------------------
>
> Key: MAPREDUCE-1172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2
> Reporter: Tsz Wo (Nicholas), SZE
>
> When I was testing HDFS-732 on 0.20, TestReduceFetch kept failing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
