Wordcount doesn't work? Which platform are you on? I just ran the wordcount on a RHEL linux machine with hadoop-0.16.0 and that worked, and my -Xmx is 512m. Could you pls share your config file (hadoop-site.xml) as well?
> -----Original Message----- > From: Stefan Groschupf [mailto:[EMAIL PROTECTED] > Sent: Sunday, March 16, 2008 12:46 PM > To: [email protected] > Subject: Re: [memory leak?] Re: MapReduce failure > > I do not instantiate 3 GB of objects, that is for sure. The > wordcount example does not run anymore so I dont think this > is something releated to my code, beside the wordcount > example many other users report the same problem: > See: > http://markmail.org/search/?q=org.apache.hadoop.mapred.MapTask > %24MapOutputBuffer.collect+order%3Adate-backward > Thanks for your help! > > Stefan > > > On Mar 15, 2008, at 11:02 PM, Devaraj Das wrote: > > > It might have something to do with your application itself. By any > > chance are you doing a lot of huge object allocation (directly or > > indirectly) within the map method? Which version of hadoop > are you on? > > > >> -----Original Message----- > >> From: Stefan Groschupf [mailto:[EMAIL PROTECTED] > >> Sent: Sunday, March 16, 2008 9:07 AM > >> To: [email protected] > >> Subject: [memory leak?] Re: MapReduce failure > >> > >> Hi there, > >> > >> we see the same situation and browsing the posts there are quite a > >> lot of people running into this OOM problem. > >> We run a own Mapper and our mapred.child.java.opts is -Xmx3048m, I > >> think that should be more then enough. > >> Also I changed io.sort.mb to 10, which had also no impact. > >> > >> Any ideas what might cause the OutOfMemoryError ? > >> Thanks. > >> Stefan > >> > >> > >> > >> > >> On Mar 9, 2008, at 10:28 PM, Amar Kamat wrote: > >> > >>> What is the heap size you are using for your tasks? Check > >>> 'mapred.child.java.opts' in your hadoop-default.xml. Try > increasing > >>> it. This will happen if you try running the random-writer + sort > >>> examples with default parameters. The maps are not able to > >> spill the > >>> data to the disk. Btw what version of HADOOP are you using? > >>> Amar > >>> On Mon, 10 Mar 2008, Ved Prakash wrote: > >>> > >>>> Hi friends, > >>>> > >>>> I have made a cluster of 3 machines, one of them is > >> master, and other > >>>> 2 slaves. I executed a mapreduce job on master but after > Map, the > >>>> execution terminates and Reduce doesn't happen. I have > checked dfs > >>>> and no output folder gets created. > >>>> > >>>> this is the error I see > >>>> > >>>> 08/03/10 10:35:21 INFO mapred.JobClient: Task Id : > >>>> task_200803101001_0001_m_000064_0, Status : FAILED > >>>> java.lang.OutOfMemoryError: Java heap space > >>>> at > >>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java > >>>> :95) > >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90) > >>>> at org.apache.hadoop.io.Text.write(Text.java:243) > >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect( > >>>> MapTask.java:347) > >>>> at org.apache.hadoop.examples.WordCount > >>>> $MapClass.map(WordCount.java > >>>> :72) > >>>> at org.apache.hadoop.examples.WordCount > >>>> $MapClass.map(WordCount.java > >>>> :59) > >>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192) > >>>> at org.apache.hadoop.mapred.TaskTracker > >>>> $Child.main(TaskTracker.java > >>>> :1787) > >>>> > >>>> 08/03/10 10:35:22 INFO mapred.JobClient: map 55% reduce > >> 17% 08/03/10 > >>>> 10:35:31 INFO mapred.JobClient: map 56% reduce 17% > >> 08/03/10 10:35:51 > >>>> INFO mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:04 INFO > >>>> mapred.JobClient: map 58% reduce 17% 08/03/10 10:36:07 INFO > >>>> mapred.JobClient: map 57% reduce 17% 08/03/10 10:36:07 INFO > >>>> mapred.JobClient: Task Id : > >>>> task_200803101001_0001_m_000071_0, Status : FAILED > >>>> java.lang.OutOfMemoryError: Java heap space > >>>> at > >>>> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java > >>>> :95) > >>>> at java.io.DataOutputStream.write(DataOutputStream.java:90) > >>>> at org.apache.hadoop.io.Text.write(Text.java:243) > >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect( > >>>> MapTask.java:347) > >>>> at org.apache.hadoop.examples.WordCount > >>>> $MapClass.map(WordCount.java > >>>> :72) > >>>> at org.apache.hadoop.examples.WordCount > >>>> $MapClass.map(WordCount.java > >>>> :59) > >>>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192) > >>>> at org.apache.hadoop.mapred.TaskTracker > >>>> $Child.main(TaskTracker.java > >>>> :1787) > >>>> > >>>> though it tries to overcome this problem but the mapreduce > >>>> application doesn't create output, can anyone tell me > why is this > >>>> happening? > >>>> > >>>> Thanks > >>>> > >>> > >> > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > >> 101tec Inc. > >> Menlo Park, California, USA > >> http://www.101tec.com > >> > >> > >> > > > > > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > 101tec Inc. > Menlo Park, California, USA > http://www.101tec.com > > >
