hi,
When I am trying to use fairscheduler as the low level scheduler for
hadoop, when I submit a job, I got NULL pointer exception in jobtracker.
I go through the hadoop code, and I think it is a bug in MesosScheduler,
Because when the job is submitted to jobtracker, the status of job is PREP,
and at this time, fairscheduler will not schedule any task of this job, so
// Let the underlying task scheduler do the actual task scheduling.
List<Task> tasks = taskScheduler.assignTasks(taskTracker);
tasks will be null. And then the following exception happnes.
13/04/17 11:30:14 INFO ipc.Server: IPC Server handler 8 on 9011, call
heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@5f0704e1, false,
false, true, 0) from 10.47.6.24:32978: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.MesosScheduler.assignTasks(MesosScheduler.java:229)
Also, I modify MesosScheduler a little bit, and it seems that the bug is
fixed.
the diff is as follow
--- a/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
+++ b/hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java
@@ -224,13 +224,14 @@ public class MesosScheduler extends TaskScheduler
implements Scheduler {
// Let the underlying task scheduler do the actual task scheduling.
List<Task> tasks = taskScheduler.assignTasks(taskTracker);
- // Keep track of which TaskTracker contains which tasks.
- for (Task task : tasks) {
- LOG.info("Assigning task : " + task.getTaskID()
- + " to tracker " + tracker);
- mesosTrackers.get(tracker).hadoopTasks.add(task.getTaskID());
+ if(task != null) {
+ // Keep track of whenich TaskTracker contains which tasks.
+ for (Task TASK_LOSTTRACKER_JVM_HEAP : tasks) {
+ LOG.info("Assigning task : " + task.getTaskID()
+ + " to tracker " + tracker);
+ mesosTrackers.get(tracker).hadoopTasks.add(task.getTaskID());
+ }
}
-
return tasks;
}
Does anybody can help me to identify this bug?
Thanks a lot.
Guodong
