Hi Amit, He means the mapred.fairscheduler.assignmultiple FairScheduler property. It is true by default, which works well for most workloads if not benchmark style workloads. I would not usually trust that as a base perf. measure of everything that comes out of an upgrade.
The other JIRA, MAPREDUCE-4451, has been resolved for 1.2.0. On Tue, Nov 27, 2012 at 3:20 PM, Amit Sela <am...@infolinks.com> wrote: > Hi Jon, > > I recently upgraded our cluster from Hadoop 0.20.3-append to Hadoop 1.0.4 > and I haven't noticed any performance issues. By "multiple assignment > feature" do you mean speculative execution > (mapred.map.tasks.speculative.execution and > mapred.reduce.tasks.speculative.execution) ? > > > On Mon, Nov 26, 2012 at 11:49 PM, Jon Allen <jayaye...@gmail.com> wrote: >> >> Problem solved, but worth warning others about. >> >> Before the upgrade the reducers for the terasort process had been evenly >> distributed around the cluster - one per task tracker in turn, looping >> around the cluster until all tasks were allocated. After the upgrade all >> reduce task had been submitted to small number of task trackers - submit >> tasks until the task tracker slots were full and then move onto the next >> task tracker. Skewing the reducers like this quite clearly hit the >> benchmark performance. >> >> The reason for this turns out to be the fair scheduler rewrite >> (MAPREDUCE-2981) that appears to have subtly modified the behaviour of the >> assign multiple property. Previously this property caused a single map and a >> single reduce task to be allocated in a task tracker heartbeat (rather than >> the default of a map or a reduce). After the upgrade it allocates as many >> tasks as there are available task slots. Turning off the multiple >> assignment feature returned the terasort to its pre-upgrade performance. >> >> I can see potential benefits to this change and need to think through the >> consequences to real world applications (though in practice we're likely to >> move away from fair scheduler due to MAPREDUCE-4451). Investigating this >> has been a pain so to warn other user is there anywhere central that can be >> used to record upgrade gotchas like this? >> >> >> On Fri, Nov 23, 2012 at 12:02 PM, Jon Allen <jayaye...@gmail.com> wrote: >>> >>> Hi, >>> >>> We've just upgraded our cluster from Hadoop 0.20.203 to 1.0.4 and have >>> hit performance problems. Before the upgrade a 15TB terasort took about 45 >>> minutes, afterwards it takes just over an hour. Looking in more detail it >>> appears the shuffle phase has increased from 20 minutes to 40 minutes. Does >>> anyone have any thoughts about what's changed between these releases that >>> may have caused this? >>> >>> The only change to the system has been to Hadoop. We moved from a >>> tarball install of 0.20.203 with all processes running as hadoop to an RPM >>> deployment of 1.0.4 with processes running as hdfs and mapred. Nothing else >>> has changed. >>> >>> As a related question, we're still running with a configuration that was >>> tuned for version 0.20.1. Are there any recommendations for tuning >>> properties that have been introduced in recent versions that are worth >>> investigating? >>> >>> Thanks, >>> Jon >> >> > -- Harsh J
