hadoop@lobster-nfs:/root$ java -d64 -version java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
On 11/15/11 4:53 AM, "Mohamed Riadh Trad" <mohamed.t...@inria.fr> wrote: > java -version > > to check if the java you are using is a 32 bits or 64 bits version. > > If you are using a 32bits version you cannot allow more than 3,5 Gbyte for the > heap. > > > Trad Mohamed Riadh, M.Sc, Ing. > PhD. student > INRIA-TELECOM PARISTECH - ENPC School of International Management > > Office: 11-15 > Phone: (33)-1 39 63 59 33 > Fax: (33)-1 39 63 56 74 > Email: riadh.t...@inria.fr <http://inria.fr/> > Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/ > <http://www-rocq.inria.fr/%7Etrad/> > > > > > Le 15 nov. 2011 à 00:39, Hoot Thompson a écrit : > >> Re: Mapreduce heap size error >> Still issues, around 2300 unique files.... >> >> >> hadoop@lobster-nfs:~/querry$ hadoop jar HadoopTest.jar -D >> mapred.child.java.opts=-Xmx4096M >> hdfs://lobster-nfs:9000/hadoop_fs/dfs/merra/seq_out >> /hadoop_fs/dfs/output/test_111114_r2.out >> 11/11/15 01:56:20 INFO hpc.Driver: Jar Name: >> /home/hadoop/querry/HadoopTest.jar >> 0 [main] INFO nccs.hpc.Driver - Jar Name: /home/hadoop/querry/HadoopTest.jar >> 0 [main] INFO nccs.hpc.Driver - Jar Name: /home/hadoop/querry/HadoopTest.jar >> 11/11/15 01:56:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing >> the arguments. Applications should implement Tool for the same. >> 60 [main] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser >> for parsing the arguments. Applications should implement Tool for the same. >> 60 [main] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser >> for parsing the arguments. Applications should implement Tool for the same. >> 11/11/15 01:56:22 INFO input.FileInputFormat: Total input paths to process : >> 2329 >> 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - >> Total input paths to process : 2329 >> 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - >> Total input paths to process : 2329 >> 11/11/15 01:56:24 INFO mapred.JobClient: Running job: job_201111150008_0003 >> 3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: >> job_201111150008_0003 >> 3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: >> job_201111150008_0003 >> 11/11/15 01:56:25 INFO mapred.JobClient: map 0% reduce 0% >> 4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% >> 4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% >> 11/11/15 01:58:03 INFO mapred.JobClient: map 1% reduce 0% >> 102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0% >> 102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0% >> 11/11/15 01:59:07 INFO mapred.JobClient: map 2% reduce 0% >> 167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0% >> 167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0% >> 11/11/15 02:00:12 INFO mapred.JobClient: map 3% reduce 0% >> 232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0% >> 232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0% >> 11/11/15 02:01:18 INFO mapred.JobClient: map 4% reduce 0% >> 298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0% >> 298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0% >> 11/11/15 02:02:21 INFO mapred.JobClient: map 5% reduce 0% >> 361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0% >> 361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0% >> 11/11/15 02:02:51 INFO mapred.JobClient: Task Id : >> attempt_201111150008_0003_r_000000_0, Status : FAILED >> 391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : >> attempt_201111150008_0003_r_000000_0, Status : FAILED >> 391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : >> attempt_201111150008_0003_r_000000_0, Status : FAILED >> Error: java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMem >> ory(ReduceTask.java:1685) >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput >> (ReduceTask.java:1545) >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(R >> educeTask.java:1394) >> at >> org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTa >> sk.java:1326) >> >> >> >> On 11/14/11 8:23 PM, "Mohamed Riadh Trad" <mohamed.t...@inria.fr >> <x-msg://1489/mohamed.t...@inria.fr> > wrote: >> >>> -D mapred.child.java.opts=-Xmx4096M > >
