Hi,all..How can I run a MR job though my own program instead of using
console to submit a job to a real Hadoop env?
I write code like this, this program works fine but i don't think it ran in
my Hadoop env,since nothing was produced in hadoop logs folder.
public int run(String[] args) throws Exception {
Configuration config = new Configuration();
config.set("fs.default.name", "hdfs://localhost:54310");
config.set("mapred.job.tracker","localhost:54311");
JobConf conf = new JobConf(config,CustomMR.class);
conf.setJobName("custom aggregation");
conf.setMapOutputKeyClass(Text.class);
conf.setMapOutputValueClass(IntWritable.class);
conf.setOutputFormat(HiveIgnoreKeyTextOutputFormat.class);
conf.setMapperClass(MapClass.class);
conf.setReducerClass(Reduce.class);
TextInputFormat.setInputPaths(conf,args[0]);
HiveIgnoreKeyTextOutputFormat.setOutputPath(conf, new
Path(args[1]));
JobClient.runJob(conf);
//conf
return 0;
}
public static void main(String[] args) throws Exception {
args = new String[]{"/user/hive/warehouse/ptest","/testout"};
int res = ToolRunner.run(new Configuration(), new CustomMR(),
args);
System.exit(res);
}
Could some one give me an example?Thanks!