Eduard, Would you try using the following properties in your job invocation?
-D mapreduce.map.java.opts=-Xmx768m -D mapreduce.reduce.java.opts=-Xmx768m -D mapreduce.map.memory.mb=2000 -D mapreduce.reduce.memory.mb=3000 Thx On Mon, Nov 5, 2012 at 7:43 AM, Kartashov, Andy <[email protected]> wrote: > Your error takes place during reduce task, when temporary files are written > to memory/disk. You are clearly running low on resources. Check your memory > “$ free –m” and disk space “$ df –H” as well as “$hadoop fs -df” > > > > I remember it took me a couple of days to figure out why I was getting heap > size error and nothing wporked! Becaue, I tried to write 7Gb output file > onto a disk (in pseudo distr mode) that only had 4Gb of free space. > > > > p.s. Always test your jobs on small input first (few lines of inputs) . > > > > p.p.s. follow your job execution through web: > http://<fully-qualified-hostan-name of your job tracker>:50030 > > > > > > From: Eduard Skaley [mailto:[email protected]] > Sent: Monday, November 05, 2012 4:10 AM > To: [email protected] > Cc: Nitin Pawar > Subject: Re: Error: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError Java Heap Space > > > > By the way it happens on Yarn not on MRv1 > > each container gets 1GB at the moment. > > can you try increasing memory per reducer ? > > > > On Wed, Oct 31, 2012 at 9:15 PM, Eduard Skaley <[email protected]> wrote: > > Hello, > > I'm getting this Error through job execution: > > 16:20:26 INFO [main] Job - map 100% reduce 46% > 16:20:27 INFO [main] Job - map 100% reduce 51% > 16:20:29 INFO [main] Job - map 100% reduce 62% > 16:20:30 INFO [main] Job - map 100% reduce 64% > 16:20:32 INFO [main] Job - Task Id : > attempt_1351680008718_0018_r_000006_0, Status : FAILED > Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error > in shuffle in fetcher#2 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:58) > at > org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:45) > at > org.apache.hadoop.mapreduce.task.reduce.MapOutput.<init>(MapOutput.java:97) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManager.unconditionalReserve(MergeManager.java:286) > at > org.apache.hadoop.mapreduce.task.reduce.MergeManager.reserve(MergeManager.java:276) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:384) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:319) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:179) > > 16:20:33 INFO [main] Job - map 100% reduce 65% > 16:20:36 INFO [main] Job - map 100% reduce 67% > 16:20:39 INFO [main] Job - map 100% reduce 69% > 16:20:41 INFO [main] Job - map 100% reduce 70% > 16:20:43 INFO [main] Job - map 100% reduce 71% > > I have no clue what the issue could be for this. I googled this issue and > checked several sources of possible solutions but nothing does fit. > > I saw this jira entry which could fit: > https://issues.apache.org/jira/browse/MAPREDUCE-4655. > > Here somebody recommends to increase the value for the property > dfs.datanode.max.xcievers / dfs.datanode.max.receiver.threads to 4096, but > this is the value for our cluster. > http://yaseminavcular.blogspot.de/2011/04/common-hadoop-hdfs-exceptions-with.html > > The issue with the to small input files doesn't fit I think, because the map > phase reads 137 files with each 130MB. Block Size is 128MB. > > The cluster uses version 2.0.0-cdh4.1.1, > 581959ba23e4af85afd8db98b7687662fe9c5f20. > > Thx > > > > > > > > > > -- > Nitin Pawar > > > > > > NOTICE: This e-mail message and any attachments are confidential, subject to > copyright and may be privileged. Any unauthorized use, copying or disclosure > is prohibited. If you are not the intended recipient, please delete and > contact the sender immediately. Please consider the environment before > printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui > l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent > être couverts par le secret professionnel. Toute utilisation, copie ou > divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire > prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. > Veuillez penser à l'environnement avant d'imprimer le présent courriel -- Alejandro
