java -version to check if the java you are using is a 32 bits or 64 bits version.
If you are using a 32bits version you cannot allow more than 3,5 Gbyte for the heap. Trad Mohamed Riadh, M.Sc, Ing. PhD. student INRIA-TELECOM PARISTECH - ENPC School of International Management Office: 11-15 Phone: (33)-1 39 63 59 33 Fax: (33)-1 39 63 56 74 Email: riadh.t...@inria.fr Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/ Le 15 nov. 2011 à 00:39, Hoot Thompson a écrit : > Still issues, around 2300 unique files.... > > > hadoop@lobster-nfs:~/querry$ hadoop jar HadoopTest.jar -D > mapred.child.java.opts=-Xmx4096M > hdfs://lobster-nfs:9000/hadoop_fs/dfs/merra/seq_out > /hadoop_fs/dfs/output/test_111114_r2.out > 11/11/15 01:56:20 INFO hpc.Driver: Jar Name: > /home/hadoop/querry/HadoopTest.jar > 0 [main] INFO nccs.hpc.Driver - Jar Name: /home/hadoop/querry/HadoopTest.jar > 0 [main] INFO nccs.hpc.Driver - Jar Name: /home/hadoop/querry/HadoopTest.jar > 11/11/15 01:56:20 WARN mapred.JobClient: Use GenericOptionsParser for parsing > the arguments. Applications should implement Tool for the same. > 60 [main] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser > for parsing the arguments. Applications should implement Tool for the same. > 60 [main] WARN org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser > for parsing the arguments. Applications should implement Tool for the same. > 11/11/15 01:56:22 INFO input.FileInputFormat: Total input paths to process : > 2329 > 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - > Total input paths to process : 2329 > 2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - > Total input paths to process : 2329 > 11/11/15 01:56:24 INFO mapred.JobClient: Running job: job_201111150008_0003 > 3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: > job_201111150008_0003 > 3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job: > job_201111150008_0003 > 11/11/15 01:56:25 INFO mapred.JobClient: map 0% reduce 0% > 4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% > 4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0% > 11/11/15 01:58:03 INFO mapred.JobClient: map 1% reduce 0% > 102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0% > 102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0% > 11/11/15 01:59:07 INFO mapred.JobClient: map 2% reduce 0% > 167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0% > 167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0% > 11/11/15 02:00:12 INFO mapred.JobClient: map 3% reduce 0% > 232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0% > 232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0% > 11/11/15 02:01:18 INFO mapred.JobClient: map 4% reduce 0% > 298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0% > 298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0% > 11/11/15 02:02:21 INFO mapred.JobClient: map 5% reduce 0% > 361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0% > 361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0% > 11/11/15 02:02:51 INFO mapred.JobClient: Task Id : > attempt_201111150008_0003_r_000000_0, Status : FAILED > 391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : > attempt_201111150008_0003_r_000000_0, Status : FAILED > 391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id : > attempt_201111150008_0003_r_000000_0, Status : FAILED > Error: java.lang.OutOfMemoryError: Java heap space > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1685) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1545) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1394) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1326) > > > > On 11/14/11 8:23 PM, "Mohamed Riadh Trad" <mohamed.t...@inria.fr> wrote: > >> -D mapred.child.java.opts=-Xmx4096M
