Hi Folks ,
I am getting the below mentioned exception while running the Map reduce job
during the copy phase of the Mappers output.
I googled about it and tried all the possible solutions suggested but none
of them worked out in my case.
I tried to increase the memory available to JVM -
D mapred.child.java.opts=-Xmx8G but didn't work out.
Then I also tried to increase the memory available for reducer -D
mapreduce.reduce.memory=2048m. No luck here also.
I also tried to reduce the -D mapred.job.reduce.input.buffer.percent , so
that it moves the output to disk instead of keeping it in memory but no
luck here also.
Please advise if I am missing something very basic here.
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
in shuffle in fetcher#5
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)
at
org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)
at
org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:411)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at
org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
Thanks
Saurabh
