+1 The change to the existing code is very limited and the perf is impressive. -C
On Fri, Sep 5, 2014 at 4:58 PM, Todd Lipcon <[email protected]> wrote: > Hi all, > > As I've reported recently [1], work on the MAPREDUCE-2841 branch has > progressed well and the development team working on it feels that it is > ready to be merged into trunk. > > For those not familiar with the JIRA (it's a bit lengthy to read from start > to finish!) the goal of this work is to build a native implementation of > the map-side sort code. The native implementation's primary advantage is > its speed: for example, terasort is 30% faster on a wall-clock basis and > 60% faster on a resource consumption basis. For clusters which make heavy > use of MapReduce, this is a substantial improvement to their efficiency. > Users may enable the feature by switching a single configuration flag, and > it will fall back to the original implementation in cases where the native > code doesn't support the configured features/types. > > The new work is entirely pluggable and off-by-default to mitigate risk. The > merge patch itself does not modify even a single line of existing code: all > necessary plug-points have already been committed to trunk for some time. > > Though we do not yet have a full +1 precommit Jenkins run on the JIRA, > there are only a few small nits to fix before merge, so I figured that we > could start the vote in parallel. Of course we will not merge until it has > a positive precommit run. > > Though this branch is a new contribution to the Apache repository, it > represents work done over several years by a large community of developers > including the following: > > Binglin Chang > Yang Dong > Sean Zhong > Manu Zhang > Zhongliang Zhu > Vincent Wang > Yan Dong > Cheng Lian > Xusen Yin > Fangqin Dai > Jiang Weihua > Gansha Wu > Avik Dey > > The vote will run for 7 days, ending Friday 9/12 EOD PST. > > I'll start the voting with my own +1. > > -Todd > > [1] > http://search-hadoop.com/m/09oay13EwlV/native+task+progress&subj=Native+task+branch+progress
