----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11119/#review20713 -----------------------------------------------------------
hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42765> thank you! hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42769> I don't understand the need for these for loops now. I completely agree with your intention to not kill a task tracker, if it was assigned a job that is not yet complete. I think this will help reduce task tracker churn. So, can't we simply now remove the job from the 'mesosTracker' and kill it if its empty? In other words, can we simply do: mesosTracker.hadoopJobs.remove(job.getJobID()); if (mesosTracker.hadoopJobs.isEmpty()) { // kill } hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42768> How is this possible, considering we got the 'completed' list from the job? hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42767> Why pull out task tracker into a temp? hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42766> s/job tasks/jobs/ ? hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/11119/#comment42770> We originally called this map 'hadoopTasks' instead of just 'tasks' to differentiate hadoop tasks from mesos tasks. With hadoop jobs, I think we can just call this map 'jobs' because there is no concept of mesos jobs. - Vinod Kone On May 14, 2013, 12:20 a.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11119/ > ----------------------------------------------------------- > > (Updated May 14, 2013, 12:20 a.m.) > > > Review request for mesos. > > > Description > ------- > > From 053d9b61f33ecbd49dba30d126a2eb9504fcba1a Mon Sep 17 00:00:00 2001 > From: Brenden Matthews <[email protected]> > Date: Mon, 13 May 2013 16:13:16 -0700 > Subject: [PATCH 12/24] Don't prematurely kill TaskTrackers. > > We assign the JobID to the internal Mesos tracker (for the TaskTracker) > rather than the TaskID. > > In the case where a TaskTracker has only map tasks assigned to it and > they have all completed, we mustn't terminate the tracker until the > entire job has finished, including the map tasks. > > Review: https://reviews.apache.org/r/11119 > --- > .../org/apache/hadoop/mapred/MesosScheduler.java | 203 > ++++++++++---------- > 1 file changed, 105 insertions(+), 98 deletions(-) > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > afe401f5265e3d9494af7eace42eec45943184a3 > > Diff: https://reviews.apache.org/r/11119/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
