Hey all,
I'm trying to chain multiple mapreduce jobs together to accomplish a
complex task. I believe that the way to do it is as follows:
JobConf conf = new JobConf(getConf(), MyClass.class);
//configure job.... set mappers, reducers, etc
SequenceFileOutputFormat.setOutputPath(conf,myPath1);
JobClient.runJob(conf);
//new job
JobConf conf2 = new JobConf(getConf(),MyClass.class)
SequenceFileInputFormat.setInputPath(conf,myPath1);
//more configuration...
JobClient.runJob(conf2)
Is this the canonical way to chain jobs? I'm having some trouble with
this method -- for especially long jobs, the latter MR tasks sometimes
do not start up.
