I apologize for asking what seems to be such a basic question, but I would use
some help with submitting a job to a remote server.
I have downloaded and installed hadoop locally in pseudo-distributed mode. I
have written some Java code to submit a job.
Here's the org.apache.hadoop.util.Tool and org.apache.hadoop.mapreduce.Mapper
I've written.
If I enable the conf.set("mapred.job.tracker", "localhost:9001") line, then I
get the exception included below.
If that line is disabled, then the job is completed. However, in reviewing the
hadoop server administration page (http://localhost:50030/jobtracker.jsp) I
don't see the job as processed by the server. Instead, I wonder if my Java code
is simply running the necessary mapper Java code, bypassing the locally
installed server.
Thanks in advance.
Alex
public class OfflineDataTool extends Configured implements Tool {
public int run(final String[] args) throws Exception {
final Configuration conf = getConf();
//conf.set("mapred.job.tracker", "localhost:9001");
final Job job = new Job(conf);
job.setJarByClass(getClass());
job.setJobName(getClass().getName());
job.setMapperClass(OfflineDataMapper.class);
job.setInputFormatClass(TextInputFormat.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job, new
org.apache.hadoop.fs.Path(args[0]));
final org.apache.hadoop.fs.Path output = new
org.apache.hadoop.fs.Path(args[1]);
FileSystem.get(conf).delete(output, true);
FileOutputFormat.setOutputPath(job, output);
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(final String[] args) {
try {
int result = ToolRunner.run(new Configuration(), new
OfflineDataTool(), new String[]{"offline/input", "offline/output"});
log.error("result = {}", result);
} catch (final Exception e) {
throw new RuntimeException(e);
}
}
}
public class OfflineDataMapper extends Mapper<LongWritable, Text, Text, Text> {
public OfflineDataMapper() {
super();
}
@Override
protected void map(final LongWritable key, final Text value, final Context
context) throws IOException, InterruptedException {
final String inputString = value.toString();
OfflineDataMapper.log.error("inputString = {}", inputString);
}
}