Hello, 

please provide me some help by giving me answers for the following 
questions:

Is my assumption right, that the within a mapred job, the number of "Map 
output records" should/must match the number "Reduce input records" when 
no combiner is used in between?

Can i screw the code in a way that the "Reduce input records" is twice the 
value of "Map output records"? And if, what could be my mistake?

What is an "input group"?

Any input is appreciated,

ud



 




[EMAIL PROTECTED] 
04/10/2008 05:54 PM
Please respond to
[email protected]


To
[email protected]
cc

Subject
Re: Counters giving double values






Hello list readers,

i'm still looking for an explanation.

maybe i put it into this way: i run a job and at the end i need to know 
how many instances of a specific type were written into an output file. 
i wanted to rely on the counters. but maybe this is not a good idea 
anyway. or what is the outcome if speculative execution is used?

how can i get the number of records contained in a MapFile - without the 
need to read it?

thank you for some input,

ud







[EMAIL PROTECTED] 
04/09/2008 10:35 AM
Please respond to
[email protected]


To
[email protected]
cc

Subject
Counters giving double values






hello,

i tried to track down the problem with wrong counter values. didnt find 
any information / cases of it. maybe it is a feature i don't understand. 
the problem is, that in a local installation (localjobrunner) i sometimes 
(somehow randomly) get wrong counter values as the log() function of the 
Counters tells me in the log:

20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Job 
complete: job_local_84
20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Counters: 21
20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Map-Reduce Framework
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map 
input records=43917
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map 
output records=43917
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map 
input bytes=2116832
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map 
output bytes=9310354
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Combine input records=0
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Combine output records=0
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Reduce input groups=83736
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Reduce input records=87834
20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - 
Reduce output records=86640

what i don't not understand is, why or how it comes, that the number of 
map output records is different than the reduce input records.
maybe somebody has a simple explanation for this effect.

i would expect that the number must be same. as you can see the number is 
(always) exactly the double value of what i expect it should be. 
and also when i check the output file, the amount of records matches the 
numbers from the map counters and not the double values.

thanks,

ud






Reply via email to