I wonder how accurate that is. However, by setting the number of reducers slightly lesser than the reduce slots, the difference acts as headroom for speculative reduce tasks. And, the goal of a single wave is also preserved.
On Mon, Apr 22, 2013 at 11:10 PM, Darpan R <[email protected]> wrote: > Hi guys, > I read somewhere that for better performance > > For maximum performance, the number of reducers should be slightly less > than > the number of reduce slots in the cluster. This allows the reducers to > finish in > one wave and fully utilizes the cluster during the reduce phase. > > I don't quite understand this, Can you please help me understand? > > Thank you. >
