[
https://issues.apache.org/jira/browse/MAPREDUCE-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13606605#comment-13606605
]
Alejandro Abdelnur commented on MAPREDUCE-5028:
-----------------------------------------------
Reverted commits from trunk and branch-2.
We need to fix test-patch, not the first time this is happening, as it is
trying to be smarter than it is by running tests only in modules that changed,
the correct thing should be to run tests in all dependent modules. Though a
change in commmon would trigger ALL tests. May be a compromise would be to run
all the tests in the corresponding component (common, hdfs, yarn, mapreduce,
tools).
> Maps fail when io.sort.mb is set to high value
> ----------------------------------------------
>
> Key: MAPREDUCE-5028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.1.1, 2.0.3-alpha, 0.23.5
> Reporter: Karthik Kambatla
> Assignee: Alejandro Abdelnur
> Priority: Critical
> Fix For: 1.2.0, 0.23.7, 2.0.5-beta
>
> Attachments: mr-5028-branch1.patch, mr-5028-branch1.patch,
> mr-5028-branch1.patch, mr-5028-trunk.patch,
> org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt
>
>
> Verified the problem exists on branch-1 with the following configuration:
> Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m,
> io.sort.mb=1280, dfs.block.size=2147483648
> Run teragen to generate 4 GB data
> Maps fail when you run wordcount on this configuration with the following
> error:
> {noformat}
> java.io.IOException: Spill failed
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
> org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
> at
> org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:375)
> at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
> at
> org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
> at
> org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
> at
> org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
