Dear fellow Hama users,
It seems that FileInputFormat.setInputPaths doesn't work correctly for
multiple HDFS paths (although it works fine in local mode using local
files). I am using Hama 0.5.0 on Hadoop 1.0.3.
I am attaching a simple code that concatenates text form files (just to
show the error).
It works fine in local mode for multiple files, it works fine in
pseudo-distributed mode for just one file, but it doesn't work in
pseudo-distributed mode on multiple HDFS files: it displays the
following with no entry in the log for this job:
&&&
hdfs://localhost:9000/user/fegaras/orders.tbl,hdfs://localhost:9000/user/fegaras/customer.tbl
12/07/28 09:42:00 INFO bsp.FileInputFormat: Total input paths to process : 2
12/07/28 09:42:00 INFO bsp.FileInputFormat: Total # of splits: 4
12/07/28 09:42:00 INFO bsp.BSPJobClient: Running job: job_201207280900_0004
12/07/28 09:42:03 INFO bsp.BSPJobClient: Current supersteps number: 0
12/07/28 09:42:03 INFO bsp.BSPJobClient: Job failed.
Both paths are correct and can be accessed separately.
Is this a Hama error or am I doing something wrong?
Thanks for your help,
Best regards
Leonidas Fegaras
U. of Texas at Arlington
import java.io.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.*;
import org.apache.hama.bsp.*;
import org.apache.hama.bsp.sync.SyncException;
import org.apache.hama.HamaConfiguration;
public class Hama1 {
public static class BSPop extends BSP<LongWritable,Text,LongWritable,Text,NullWritable> {
@Override
public void bsp ( BSPPeer<LongWritable,Text,LongWritable,Text,NullWritable> peer )
throws IOException, SyncException, InterruptedException {
LongWritable key = new LongWritable();
Text data = new Text();
while (peer.readNext(key,data))
peer.write(key,data);
}
}
public static void main ( String[] args ) throws Exception {
HamaConfiguration conf = new HamaConfiguration();
BSPJob job = new BSPJob(conf,Hama1.class);
job.setBspClass(BSPop.class);
Path outpath = new Path("out");
job.setOutputPath(outpath);
job.setOutputKeyClass(LongWritable.class);
job.setOutputValueClass(TextOutputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
job.setInputFormat(TextInputFormat.class);
job.setInputKeyClass(LongWritable.class);
job.setInputValueClass(Text.class);
FileInputFormat.setInputPaths(job,"orders.tbl,customer.tbl");
System.out.println("&&& "+conf.get("bsp.input.dir"));
job.waitForCompletion(true);
}
}
