Hi Francis,
Could you check whether those configuration files are getting
loaded or not, There could be a chance that these configuration files are not
getting loaded into configuration object due to some invalid path reason.
conf1.addResource("resources/mapred-site.xml");
// Can you try printing the properties 'yarn.resourcemanager.address' &
'mapreduce.framework.name' values and check they are coming as per the config
files
JobConf conf = new JobConf(conf1);
Thanks
Devaraj k
From: Azuryy Yu [mailto:[email protected]]
Sent: 10 July 2013 16:22
To: [email protected]
Subject: Re: cannot submit a job via java client in hadoop- 2.0.5-alpha
you didn't set yarn.nodemanager.address in your yarn-site.xml
On Wed, Jul 10, 2013 at 4:33 PM, Francis.Hu
<[email protected]<mailto:[email protected]>> wrote:
Hi,All
I have a hadoop- 2.0.5-alpha cluster with 3 data nodes . I have Resource
Manager and all data nodes started and can access web ui of Resource Manager.
I wrote a java client to submit a job as TestJob class below. But the job is
never submitted successfully. It throws out exception all the time.
My configurations are attached. Can anyone help me? Thanks.
---------my-java client
public class TestJob {
public void execute() {
Configuration conf1 = new Configuration();
conf1.addResource("resources/core-site.xml");
conf1.addResource("resources/hdfs-site.xml");
conf1.addResource("resources/yarn-site.xml");
conf1.addResource("resources/mapred-site.xml");
JobConf conf = new JobConf(conf1);
conf.setJar("/home/francis/hadoop-jobs/MapReduceJob.jar");
conf.setJobName("Test");
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(DisplayRequestMapper.class);
conf.setReducerClass(DisplayRequestReducer.class);
FileInputFormat.setInputPaths(conf,new
Path("/home/francis/hadoop-jobs/2013070907.FNODE.2.txt"));
FileOutputFormat.setOutputPath(conf, new
Path("/home/francis/hadoop-jobs/result/"));
try {
JobClient client = new JobClient(conf);
RunningJob job = client.submitJob(conf);
job.waitForCompletion();
} catch (IOException e) {
e.printStackTrace();
}
}
}
----------Exception
jvm 1 | java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name<http://mapreduce.framework.name> and
the correspond server addresses.
jvm 1 | at
org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:119)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:81)
jvm 1 | at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:74)
jvm 1 | at org.apache.hadoop.mapred.JobClient.init(JobClient.java:482)
jvm 1 | at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:461)
jvm 1 | at com.rh.elastic.hadoop.job.TestJob.execute(TestJob.java:59)
Thanks,
Francis.Hu
