[
https://issues.apache.org/jira/browse/MAPREDUCE-4933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551532#comment-13551532
]
Robert Joseph Evans commented on MAPREDUCE-4933:
------------------------------------------------
OK I have dug into this a bit more and I am not as scared as I was previously.
I could definitely be wrong so if someone could confirm my analysis on this I
would appreciate it.
So there are two places where we could get issues. The first is because we get
the file size after writing and without closing the file. This is not a huge
issue because that size is only used to approximate how much space will be
needed for the final merge file, which we just wrote out so who cares.
The second case is a race between writing the merge data out to a file and
reading it back in. This will only happen on the final merge when the reducer
has too much buffered in memory and we need to spill to disk (we exceeded
mapred.job.reduce.input.buffer.percent). But even then it becomes a race
between java's buffered IO getting pushed out to disk and another part of the
code reading it back in. I have a hard time believing that we are going to
lose that race ever. i think we can move the back to major, sorry about
scaring everyone.
> MR1 final merge asks for length of file it just wrote before flushing it
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-4933
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4933
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, task
> Affects Versions: 1.1.1
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Priority: Blocker
> Attachments: MAPREDUCE-4933-branch-1.patch
>
>
> createKVIterator in ReduceTask contains the following code:
> {code}
> try {
> Merger.writeFile(rIter, writer, reporter, job);
> addToMapOutputFilesOnDisk(fs.getFileStatus(outputPath));
> } catch (Exception e) {
> if (null != outputPath) {
> fs.delete(outputPath, true);
> }
> throw new IOException("Final merge failed", e);
> } finally {
> if (null != writer) {
> writer.close();
> }
> }
> {code}
> Merger#writeFile() does not close the file after writing it, so when
> fs.getFileStatus() is called on it, it may not return the correct length.
> This causes bad accounting further down the line, which can lead to map
> output data being lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
