@Kishore, Agreed but but shouldn't 'Reduce shuffle bytes' count decrease with the use of Combiners?
Regards, Shahab On Fri, May 10, 2013 at 2:00 PM, Kishore <[email protected]> wrote: > Combiner will be used between mapper and reduce, so the mapper output for > both with used combiner and without used combiner are same. > > Thanks, > Kishore. > > Sent from my iPhone > > On 10-May-2013, at 8:49 PM, Han JU <[email protected]> wrote: > > Hi, > > For a MapReduce job with lots of intermediate results between mapper and > reducer, I implement a combiner function with a more compact representation > of the result data and I verified the final result is good when using > combiner. But when I look at the job counter "FILE_BYTES_WRITTEN" or > "Reduce shuffle bytes", the number with combiner is twice bigger than > without combiner. In my comprehension, these two counters represent the > output size of mapper. And with a combiner, the size of mapper output > should decrease, but it's not the case here. > > So it means that my combiner doesn't work and it actually increase the > size of mapper output? > > Thanks! > -- > *JU Han* > > Software Engineer Intern @ KXEN Inc. > UTC - Université de Technologie de Compiègne > * **GI06 - Fouille de Données et Décisionnel* > > +33 0619608888 > >
