[
https://issues.apache.org/jira/browse/HADOOP-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594858#action_12594858
]
devaraj edited comment on HADOOP-3356 at 5/7/08 4:39 AM:
-------------------------------------------------------------
This part of the code must never be hit under normal circumstances for
intermediate merges (during shuffle). We should only do single-level merges for
the intermediate merges. I chatted with Arun offline and he agreed on this.
Note that this part works as expected when it is supposed to be executed - for
multi-level merges and that happens only at the end of the shuffle (when the fs
is the localfs).
We probably should fix this for completeness sake but it is definitely not a
critical/major issue.
was (Author: devaraj):
This part of the code must never be hit under normal circumstances for
intermediate merges (during shuffle). I chatted with Arun offline and he agreed
on this.
> SequenceFile.MergeQueue.merge inadvertently creates merge-outputs in the
> wrong FileSystem, at times in the InMemoryFileSystem
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3356
> URL: https://issues.apache.org/jira/browse/HADOOP-3356
> Project: Hadoop Core
> Issue Type: Bug
> Components: io
> Affects Versions: 0.16.3
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Priority: Minor
> Fix For: 0.18.0
>
>
> The offending code is:
> {code:title=SequenceFile.java}
> Path outputFile = lDirAlloc.getLocalPathForWrite(
> tmpFilename.toString(),
> approxOutputSize, conf);
> LOG.debug("writing intermediate results to " + outputFile);
> Writer writer = cloneFileAttributes(
>
> fs.makeQualified(segmentsToMerge.get(0).segmentPathName),
> fs.makeQualified(outputFile),
> null);
> {code}
> *fs* is InMemoryFileSystem when ReduceTask.ReduceCopier constructs it... so
> the wrong FileSystem is used during intermediate merges.
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
