Still issues, around 2300 unique files....
hadoop@lobster-nfs:~/querry$ hadoop jar HadoopTest.jar -D
mapred.child.java.opts=-Xmx4096M
hdfs://lobster-nfs:9000/hadoop_fs/dfs/merra/seq_out
/hadoop_fs/dfs/output/test_111114_r2.out
11/11/15 01:56:20 INFO hpc.Driver: Jar Name:
/home/hadoop/querry/HadoopTest.jar
0 [main] INFO nccs.hpc.Driver - Jar Name:
/home/hadoop/querry/HadoopTest.jar
0 [main] INFO nccs.hpc.Driver - Jar Name:
/home/hadoop/querry/HadoopTest.jar
11/11/15 01:56:20 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
60 [main] WARN org.apache.hadoop.mapred.JobClient - Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
60 [main] WARN org.apache.hadoop.mapred.JobClient - Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
11/11/15 01:56:22 INFO input.FileInputFormat: Total input paths to process :
2329
2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat -
Total input paths to process : 2329
2154 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat -
Total input paths to process : 2329
11/11/15 01:56:24 INFO mapred.JobClient: Running job: job_201111150008_0003
3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
job_201111150008_0003
3693 [main] INFO org.apache.hadoop.mapred.JobClient - Running job:
job_201111150008_0003
11/11/15 01:56:25 INFO mapred.JobClient: map 0% reduce 0%
4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
4696 [main] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
11/11/15 01:58:03 INFO mapred.JobClient: map 1% reduce 0%
102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0%
102916 [main] INFO org.apache.hadoop.mapred.JobClient - map 1% reduce 0%
11/11/15 01:59:07 INFO mapred.JobClient: map 2% reduce 0%
167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0%
167042 [main] INFO org.apache.hadoop.mapred.JobClient - map 2% reduce 0%
11/11/15 02:00:12 INFO mapred.JobClient: map 3% reduce 0%
232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0%
232168 [main] INFO org.apache.hadoop.mapred.JobClient - map 3% reduce 0%
11/11/15 02:01:18 INFO mapred.JobClient: map 4% reduce 0%
298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0%
298290 [main] INFO org.apache.hadoop.mapred.JobClient - map 4% reduce 0%
11/11/15 02:02:21 INFO mapred.JobClient: map 5% reduce 0%
361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0%
361418 [main] INFO org.apache.hadoop.mapred.JobClient - map 5% reduce 0%
11/11/15 02:02:51 INFO mapred.JobClient: Task Id :
attempt_201111150008_0003_r_000000_0, Status : FAILED
391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
attempt_201111150008_0003_r_000000_0, Status : FAILED
391477 [main] INFO org.apache.hadoop.mapred.JobClient - Task Id :
attempt_201111150008_0003_r_000000_0, Status : FAILED
Error: java.lang.OutOfMemoryError: Java heap space
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1685)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1545)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1394)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1326)
On 11/14/11 8:23 PM, "Mohamed Riadh Trad" <mohamed.t...@inria.fr> wrote:
> -D mapred.child.java.opts=-Xmx4096M
