Hello, please provide me some help by giving me answers for the following questions:
Is my assumption right, that the within a mapred job, the number of "Map output records" should/must match the number "Reduce input records" when no combiner is used in between? Can i screw the code in a way that the "Reduce input records" is twice the value of "Map output records"? And if, what could be my mistake? What is an "input group"? Any input is appreciated, ud [EMAIL PROTECTED] 04/10/2008 05:54 PM Please respond to [email protected] To [email protected] cc Subject Re: Counters giving double values Hello list readers, i'm still looking for an explanation. maybe i put it into this way: i run a job and at the end i need to know how many instances of a specific type were written into an output file. i wanted to rely on the counters. but maybe this is not a good idea anyway. or what is the outcome if speculative execution is used? how can i get the number of records contained in a MapFile - without the need to read it? thank you for some input, ud [EMAIL PROTECTED] 04/09/2008 10:35 AM Please respond to [email protected] To [email protected] cc Subject Counters giving double values hello, i tried to track down the problem with wrong counter values. didnt find any information / cases of it. maybe it is a feature i don't understand. the problem is, that in a local installation (localjobrunner) i sometimes (somehow randomly) get wrong counter values as the log() function of the Counters tells me in the log: 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Job complete: job_local_84 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Counters: 21 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map-Reduce Framework 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map input records=43917 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map output records=43917 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map input bytes=2116832 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Map output bytes=9310354 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Combine input records=0 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Combine output records=0 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Reduce input groups=83736 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Reduce input records=87834 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient - Reduce output records=86640 what i don't not understand is, why or how it comes, that the number of map output records is different than the reduce input records. maybe somebody has a simple explanation for this effect. i would expect that the number must be same. as you can see the number is (always) exactly the double value of what i expect it should be. and also when i check the output file, the amount of records matches the numbers from the map counters and not the double values. thanks, ud
