[
https://issues.apache.org/jira/browse/MAPREDUCE-6447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040315#comment-16040315
]
Adam Sotona commented on MAPREDUCE-6447:
----------------------------------------
Hi guys,
I'm experiencing exactly the same issue, even I decreased the values following
way:
mapreduce.reduce.shuffle.input.buffer.percent=0.70
mapreduce.reduce.shuffle.memory.limit.percent=0.2
mapreduce.reduce.shuffle.parallelcopies=4
According to the above calculations 0.7 * 0.2 * 4 = 0.56 < 1
Could someone explain it?
I'm running very long MapReduce process, where some of the reducers almost
deterministically fail on OOME, repeated attempts still cause OOME and the
whole restarted application with reduced shuffle parameters still fails (after
several days of computation).
Is there any reason why InMemoryMapOutput blindly allocates memory despite the
actual situation?
Could Fetcher.copyMapOutput recover from that situation by falling into an
alternative MapOutput?
Thanks
> reduce shuffle throws "java.lang.OutOfMemoryError: Java heap space"
> -------------------------------------------------------------------
>
> Key: MAPREDUCE-6447
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6447
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.5.0, 2.6.0, 2.5.1, 2.7.1
> Reporter: shuzhangyao
> Assignee: shuzhangyao
> Priority: Minor
>
> 2015-08-11 14:03:54,550 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child :
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#10
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
> at
> org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
> at
> org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
> at
> org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:303)
> at
> org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:293)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:329)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
