[
https://issues.apache.org/jira/browse/MAPREDUCE-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234461#comment-14234461
]
Eric Payne commented on MAPREDUCE-6166:
---------------------------------------
Thanks [~jira.shegalov], for taking time to investigate this issue.
The unit tests are not catching this. I am testing this in a 10-node secure
cluster.
I am running wordcount on a file that 1) has no repeated words and 2) is large
enough to ensure that at least some of the map outputs are shuffled to disk:
{code}
$ $HADOOP_PREFIX/bin/hadoop fs -cat
Input/NoRecurringWords/NoRecurringWords-part0.txt | wc -l -w
4008920 4008920
$ $HADOOP_PREFIX/bin/hadoop jar
$HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-examples-$HADOOP_VERSION.jar
wordcount Input/NoRecurringWords/NoRecurringWords-part0.txt Output/01
{code}
If I implement the fix by adjusting {{bytesLeft}} and leaving {{input.read()}}
alone, the {{OnDiskMapOutput#shuffle}} does succeed and writes the map output
to disk in a temporary location. However, when the {{Merger}} goes to read that
temporary file (via {{RawKVIteratorReader}} in {{MergeManager}}) , it fails
with the following exception:
{code}
2014-12-04 17:47:12,040 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : org.apache.hadoop.fs.ChecksumException: Checksum Error
at
org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:228)
at
org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:152)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:127)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at
org.apache.hadoop.io.IOUtils.wrappedReadForCompressedData(IOUtils.java:170)
at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:378)
at org.apache.hadoop.mapred.IFile$Reader.nextRawKey(IFile.java:426)
at org.apache.hadoop.mapred.Merger$Segment.nextRawKey(Merger.java:337)
at
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java:519)
at org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:547)
at org.apache.hadoop.mapred.ReduceTask$4.next(ReduceTask.java:601)
...
{code}
This is because the checksum is not on the end of the temporary file.
On the other hand, if I leave {{bytesLeft}} alone and instead call
{{((IFileInputStream)input).readWithChecksum}}, the reducers all succeed. This
is because {{readWithChecksum}} not only compares the input against the
checksum, it also includes the checksum at the end of the byte buffer.
Please let me know if that makes sense.
> Reducers do not catch bad map output transfers during shuffle if data
> shuffled directly to disk
> -----------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6166
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6166
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.6.0
> Reporter: Eric Payne
> Assignee: Eric Payne
> Attachments: MAPREDUCE-6166.v1.201411221941.txt,
> MAPREDUCE-6166.v2.201411251627.txt
>
>
> In very large map/reduce jobs (50000 maps, 2500 reducers), the intermediate
> map partition output gets corrupted on disk on the map side. If this
> corrupted map output is too large to shuffle in memory, the reducer streams
> it to disk without validating the checksum. In jobs this large, it could take
> hours before the reducer finally tries to read the corrupted file and fails.
> Since retries of the failed reduce attempt will also take hours, this delay
> in discovering the failure is multiplied greatly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
