Please use the web UI to check the cluster capacity. I think your cluster is not working correctly now, or you didn't restart your cluster.
On Sun, Mar 9, 2014 at 2:00 AM, Ammar Sahib <[email protected]> wrote: > Hi > > I tried to increase the bsp.tasks.maximum in hama-default.xml from 3 to 30 > but I still get the same error. > I am thinking of of reducing the number of blocks of the input file by > controlling the parameter dfs.namenode.fs-limits.min-block-size in > hdfs-default.xml. Do you think this might be a good approach that might > solve the problem? > > > > On Friday, March 7, 2014 11:32 PM, Edward J. Yoon <[email protected]> > wrote: > If the number of blocks of input file is 52 (see [1]), you should > increase number of task slots by adding new machine or increasing max > tasks number per node "bsp.tasks.maximum". > > 1. > http://stackoverflow.com/questions/11168427/viewing-the-number-of-blocks-for-a-file-in-hadoop > > On Sat, Mar 8, 2014 at 12:22 AM, Ammar Sahib <[email protected]> wrote: >> Hi >> >> >> I am using HAMA 0.6.4 and I am running my custom program using a cluster >> of 4 machines. My input is a single file and I am setting the number of >> BSP tasks to the number of Groom servers by using >> (JOB.setNumBspTask(cluster.getGroomServers()). I am using the >> HashPartitioner.class to partition the data. >> >> >> I have a problem when I load my data. When I run my custom program I get >> the following error messages: >> >> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process : >> 1 >> 14/03/07 16:02:34 INFO util.NativeCodeLoader: Loaded the >> native-hadoop library >> 14/03/07 16:02:34 WARN snappy.LoadSnappy: Snappy native library not loaded >> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process : >> 1 >> Exception >> in thread "main" java.io.IOException: Job failed! The number of splits >> has exceeded the number of max tasks. The number of splits: 52, The >> number of max tasks: 20 >> at >> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:349) >> at >> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) >> at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) >> at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) >> at >> org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:460) >> >> at >> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:341) >> at >> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) >> at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) >> at org.apache.hama.graph.GraphJob.submit(GraphJob.java:208) >> at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) >> at >> de.rwthaachen.dbis.i5cloudmatch.controller.Matcher.main(Matcher.java:479) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> >> Any advice of how to solve this problem? >> >> RegardsAmmar > > > > -- > Edward J. Yoon (@eddieyoon) > Chief Executive Officer > DataSayer, Inc. > > -- Edward J. Yoon (@eddieyoon) Chief Executive Officer DataSayer, Inc.
