[
https://issues.apache.org/jira/browse/MAPREDUCE-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13747149#comment-13747149
]
Omkar Vinit Joshi commented on MAPREDUCE-5472:
----------------------------------------------
I think we should try running same job with 100 number of reducers "-r 100"
instead of default 1. In the Application master UI we will see the number of
reducers which actually finished. After RM restart It should start from there
instead of 0. However, I was not able to reproduce
*Error: java.io.IOException: Error while reading compressed data at
org.apache.hadoop.io.IOUtils.wrappedReadForCompressedData(IOUtils.java:174) at*
> reducer of sort job restarts from scratch in between after RM restart
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-5472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5472
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: yeshavora
> Assignee: Omkar Vinit Joshi
> Priority: Blocker
>
> Steps Followed:
> 1) Run a sort job. As soon as it finishes all the map tasks. [100% map],
> restart resource manager.
> 2) Analyse the progress of the sort job.
> It starts with 100% map 0% reduce
> 100% map 32% reduce
> 100% map 0% reduce
> Reducer stays at 30% reduce for around 5-10 minutes. and again start reducer
> from scratch.
> Log from failed reducer attempt:
> Error: java.io.IOException: Error while reading compressed data at
> org.apache.hadoop.io.IOUtils.wrappedReadForCompressedData(IOUtils.java:174)
> at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:383) at
> org.apache.hadoop.mapred.IFile$Reader.nextRawValue(IFile.java:444) at
> org.apache.hadoop.mapred.Merger$Segment.nextRawValue(Merger.java:327) at
> org.apache.hadoop.mapred.Merger$Segment.getValue(Merger.java:309) at
> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:533) at
> org.apache.hadoop.mapred.ReduceTask$4.next(ReduceTask.java:619) at
> org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:154)
> at
> org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:121)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:297)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:170) at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:645) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:405) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:396) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by:
> org.apache.hadoop.fs.FSError: java.io.IOException: Input/output error at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:177)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at
> java.io.BufferedInputStream.read(BufferedInputStream.java:317) at
> java.io.DataInputStream.read(DataInputStream.java:132) at
> org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:209)
> at org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:152)
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:127)
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> at
> org.apache.hadoop.io.IOUtils.wrappedReadForCompressedData(IOUtils.java:170)
> ... 17 more Caused by: java.io.IOException: Input/output error at
> java.io.FileInputStream.readBytes(Native Method) at
> java.io.FileInputStream.read(FileInputStream.java:220) at
> org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.read(RawLocalFileSystem.java:110)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:171)
> ... 26 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
