----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10918/#review20326 -----------------------------------------------------------
Thanks Brenden! It seems there are a few independent changes here: -More accurate computation of pending map / reduce tasks. -Treating unhealthy nodes as non-idle. -Scheduling smaller TaskTrackers when possible. We could get these committed for you faster if these changes were sent out separately, support/post-reviews.py will make this really easy! :) :) hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/10918/#comment41865> Can you elaborate as to what you were seeing here? Were the values stale, or incorrect, or something else? hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/10918/#comment41866> Thank you, a few questions here: 1. What does "unhealthy" mean in Hadoop terms? 2. Is there any action we should take for nodes that are unhealthy? Do we want to kill them? Give them time? Something else? hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java <https://reviews.apache.org/r/10918/#comment41867> This is a bit tricky to decipher, what is the end goal here, schedule the biggest TaskTracker that fits in the offer? Do you want to be allowing TaskTrackers with a different number of map / reduce slots? (e.g. 1 map, 2 reduce slots). - Ben Mahler On May 3, 2013, 6:44 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10918/ > ----------------------------------------------------------- > > (Updated May 3, 2013, 6:44 p.m.) > > > Review request for mesos. > > > Description > ------- > > From d0804b2dccfc6c8f8bf1b86934be219acd72e8f2 Mon Sep 17 00:00:00 2001 > From: Brenden Matthews <[email protected]> > Date: Fri, 26 Apr 2013 13:02:06 -0700 > Subject: [PATCH 1/9] Improve MesosScheduler for Hadoop. > > We'll try to allocate map/reduce slots in such a way that we ensure we > utilize the entire cluster effectively. We'll use up to the maximum > desired number of map/reduce slots, starting from a minimum of 1 map or > 1 reduce slot. > --- > .../org/apache/hadoop/mapred/MesosScheduler.java | 112 > +++++++++++++++++--- > 1 file changed, 95 insertions(+), 17 deletions(-) > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java afe401f > > Diff: https://reviews.apache.org/r/10918/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
