Lijie Xu created MAPREDUCE-5482:
-----------------------------------
Summary: Release kvoffsets and kvindices before going to merge
phase
Key: MAPREDUCE-5482
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5482
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 1.2.0
Environment: Any Environment
Reporter: Lijie Xu
In org.apache.hadoop.mapred.MapTask.MapOutputBuffer.flush() method, we only
released the kvbuffer before the merge phase (i.e., mergeParts()). Though
kvindices and kvoffsets are small in many cases, we should release them
theoretically before mergeParts().
try {
spillThread.interrupt();
spillThread.join();
} catch (InterruptedException e) {
throw (IOException)new IOException("Spill failed"
).initCause(e);
}
// release sort buffer before the merge
kvbuffer = null;
mergeParts();
Path outputPath = mapOutputFile.getOutputFile();
fileOutputByteCounter.increment(rfs.getFileStatus(outputPath).getLen());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
