Hi,
Well I've figured some of it out. I needed to initialise the new
JobClient after setting its configuration. So the code looks like:
JobClient aJC = new JobClient();
String jobid = jobConf.get("mapred.job.id");
aJC.setConf(jobConf);
aJC.init();
This works fine in the Map function for killing the job. However, I'm
still getting an exception in the launch function, on the
getMapTasksReports() line, albeit at a deeper level. THe exception I
get is:
org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.NullPointerException
at java.util.TreeMap.compare(TreeMap.java:1093)
at java.util.TreeMap.getEntry(TreeMap.java:347)
at java.util.TreeMap.get(TreeMap.java:265)
at
org.apache.hadoop.mapred.JobTracker.getMapTaskReports(JobTracker.java:1522)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:341)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:573)
at org.apache.hadoop.ipc.Client.call(Client.java:471)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:165)
at $Proxy1.getMapTaskReports(Unknown Source)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:505)
at imperial.oliverhaggarty.RTAMapRed1a.launch(RTAMapRed1a.java:344)
at imperial.oliverhaggarty.RTAMapRed1a.main(RTAMapRed1a.java:695)
I've called runJob() in a separate thread, as I thought once the job
had finished the set of TaskReports might get deleted. I've also put a
sleep in before I start checking for TaskReports to give time for
everything to initialise. I've copied the code below. Hopefully one of
you Hadoop guru's can point me in the right direction!
Cheers,
Ollie
Code:
Thread t = new Thread() {
public void run() {
System.out.println("In thread");
try {
JobClient.runJob(jobConf);
}
catch(IOException e) {
e.printStackTrace();
}
}
};
t.start();
System.out.println("Afterrunningthread");
try {
Thread.sleep(20000);
}
catch(InterruptedException e) {
e.printStackTrace();
}
JobClient aJC = new JobClient();
String jobid = jobConf.get("mapred.job.id");
aJC.setConf(jobConf);
aJC.init();
TaskReport [] treps = null;
do {
try {
/*GetExcetpion here-->*/ treps = aJC.getMapTaskReports(jobid);
}
catch(Exception e) {
e.printStackTrace();
break;
}
for(TaskReport trep : treps) {
System.out.println(trep.getState());
}
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
while(treps != null && treps.length > 0);
Quoting [EMAIL PROTECTED]:
Hi Michael,
Thanks for the reply. I've tried to write some code to do this now but
its not working. I was wondering if there's anything obviously wrong?
After my runJob() I put (just as a test):
JobClient aJC = new JobClient();
String jobid = jobConf.get("mapred.job.id");
aJC.setConf(jobConf); //I've tried with and without this line
TaskReport [] treps = aJC.getMapTaskReports(jobid);
for(TaskReport trep : treps) {
System.out.println(trep.getState());
}
However, when I run it, I get a NullPointerException on the
aJC.getMapTasksReports() line. I know its getting the correct jobid.
I've also tried similar code in my Map function for killing the class,
but I get a NullPointerException when I try and do:
RunningJob rj = aJC.getJob(jobid);
I'm thinking the new JobClient class needs to be connected to the
JobTracker in some way? If so, could someone explain how this is done?
Or am I way off?
Thanks,
Ollie
Quoting Michael Bieniosek <[EMAIL PROTECTED]>:
On 8/2/07 5:20 AM, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
I've found the
getMapTaskReports method in the JobClient class, but can't work out
how to access it other than by creating a new instance of JobClient -
but then that JobClient would be a differnt one to the one that was
running my job, so would access a different set of TaskReports?
That doesn't matter -- jobs are bound to jobtrackers, not to jobclients.
You can create a new JobClient and access all the jobs that the jobtracker
knows about.
-Michael
