Input group refers to the number of <key, {set-of-values}> that the reducer
gets. So if your map output looked like
<K, 1>
<K, 1>
To the reducer, the number of input groups would be 1 - <K, {1,1}> and the
number of records would be 2.
The fact that the number of reduce input records doubled beats me ... Could
you please let us know the map output key/value types and if the types are
not the hadoop built-in types, pls also paste the code for
serializing/deserializing them.
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, April 16, 2008 12:41 PM
> To: [email protected]
> Subject: Re: Counters giving double values
>
> Hello,
>
> please provide me some help by giving me answers for the following
> questions:
>
> Is my assumption right, that the within a mapred job, the
> number of "Map output records" should/must match the number
> "Reduce input records" when no combiner is used in between?
>
> Can i screw the code in a way that the "Reduce input records"
> is twice the value of "Map output records"? And if, what
> could be my mistake?
>
> What is an "input group"?
>
> Any input is appreciated,
>
> ud
>
>
>
>
>
>
>
>
> [EMAIL PROTECTED]
> 04/10/2008 05:54 PM
> Please respond to
> [email protected]
>
>
> To
> [email protected]
> cc
>
> Subject
> Re: Counters giving double values
>
>
>
>
>
>
> Hello list readers,
>
> i'm still looking for an explanation.
>
> maybe i put it into this way: i run a job and at the end i
> need to know
> how many instances of a specific type were written into an
> output file.
> i wanted to rely on the counters. but maybe this is not a good idea
> anyway. or what is the outcome if speculative execution is used?
>
> how can i get the number of records contained in a MapFile -
> without the
> need to read it?
>
> thank you for some input,
>
> ud
>
>
>
>
>
>
>
> [EMAIL PROTECTED]
> 04/09/2008 10:35 AM
> Please respond to
> [email protected]
>
>
> To
> [email protected]
> cc
>
> Subject
> Counters giving double values
>
>
>
>
>
>
> hello,
>
> i tried to track down the problem with wrong counter values.
> didnt find
> any information / cases of it. maybe it is a feature i don't
> understand.
> the problem is, that in a local installation (localjobrunner)
> i sometimes
> (somehow randomly) get wrong counter values as the log()
> function of the
> Counters tells me in the log:
>
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Job
> complete: job_local_84
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Counters: 21
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Map-Reduce Framework
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> input records=43917
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> output records=43917
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> input bytes=2116832
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> output bytes=9310354
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Combine input records=0
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Combine output records=0
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce input groups=83736
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce input records=87834
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce output records=86640
>
> what i don't not understand is, why or how it comes, that the
> number of
> map output records is different than the reduce input records.
> maybe somebody has a simple explanation for this effect.
>
> i would expect that the number must be same. as you can see
> the number is
> (always) exactly the double value of what i expect it should be.
> and also when i check the output file, the amount of records
> matches the
> numbers from the map counters and not the double values.
>
> thanks,
>
> ud
>
>
>
>
>
>
>