[ https://issues.apache.org/jira/browse/MAPREDUCE-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza resolved MAPREDUCE-4933. ----------------------------------- Resolution: Fixed > MR1 final merge asks for length of file it just wrote before flushing it > ------------------------------------------------------------------------ > > Key: MAPREDUCE-4933 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4933 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, task > Affects Versions: 1.1.1 > Reporter: Sandy Ryza > Assignee: Sandy Ryza > Attachments: MAPREDUCE-4933-branch-1.patch > > > createKVIterator in ReduceTask contains the following code: > {code} > try { > Merger.writeFile(rIter, writer, reporter, job); > addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath)); > } catch (Exception e) { > if (null != outputPath) { > fs.delete(outputPath, true); > } > throw new IOException("Final merge failed", e); > } finally { > if (null != writer) { > writer.close(); > } > } > {code} > Merger#writeFile() does not close the file after writing it, so when > fs.getFileStatus() is called on it, it may not return the correct length. > This causes bad accounting further down the line, which can lead to map > output data being lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira