Thanks so far.
key and values are custom implementations.
key implements WritableComparable
value extends VersionedWritable
btw. The only problem i encounter is that the Counter values are wrong. If
i check the records in the MapFile (re-read it) which is written as the
output of the mapred job, the amount of records is correct and represents
the halve of the reported counter value.
the same applies for the results of the operations which are carried out
in the reduce(). everything is correct, except the counter values.
the whole thing happens only sometimes.
Key serializing / Deserializing, i guess you want to see this part of the
code:
public int language;
public String term;
public void readFields(DataInput in) throws IOException {
language = in.readInt();
term = Text.readString(in).toString();
}
public void write(DataOutput out) throws IOException {
out.writeInt(language);
Text.writeString(out, term);
}
i runned a mapred job sequencially over a defined set of data. the
following data represents statistics of the outcome (s specific counter
value). (horizontally you see the outcome of the different runs on the
identical data)
what you can see here, is, that the job behaves differently in every run
and that it has some data sets where the wrong (double) counter values
likes to show up (sometimes).
1606742
1606742
1606742
803371
1606742
803371
770238
770238
770238
770238
770238
770238
743497
743497
743497
743497
743497
743497
788210
788210
788210
788210
788210
788210
765884
765884
765884
765884
765884
765884
784747
784747
784747
784747
784747
784747
760986
760986
760986
760986
760986
760986
746474
746474
746474
746474
746474
746474
742191
742191
742191
742191
742191
742191
716320
716320
716320
716320
716320
716320
696199
696199
696199
696199
696199
696199
732110
732110
732110
732110
732110
732110
739052
739052
739052
739052
739052
739052
756332
756332
756332
756332
756332
756332
744607
744607
744607
744607
744607
744607
748204
748204
748204
748204
748204
748204
736882
736882
736882
736882
736882
736882
700750
700750
700750
700750
700750
700750
259806
259806
259806
259806
259806
259806
1279252
639626
1279252
1279252
1279252
1279252
1270780
1270780
1270780
1270780
1270780
1270780
524832
524832
524832
524832
524832
524832
520642
520642
520642
520642
520642
520642
534722
534722
534722
534722
534722
534722
561428
1122856
1122856
561428
561428
1122856
535732
535732
535732
535732
535732
535732
525191
525191
525191
525191
525191
525191
515451
515451
515451
515451
515451
515451
531535
531535
531535
531535
531535
531535
502990
502990
502990
502990
502990
502990
515943
515943
515943
515943
515943
515943
523106
523106
523106
523106
523106
523106
510348
510348
510348
510348
510348
510348
535886
535886
535886
535886
535886
535886
522792
522792
522792
522792
522792
522792
535406
535406
535406
535406
535406
535406
523322
1046644
523322
523322
523322
523322
546711
546711
546711
546711
546711
546711
565917
565917
1131834
1131834
565917
1131834
564474
1128948
1128948
1128948
564474
1128948
1148090
1148090
1148090
1148090
1148090
1148090
565539
1131078
1131078
565539
565539
1131078
567117
1134234
1134234
567117
1134234
1134234
549599
549599
1099198
1099198
549599
549599
574316
1148632
1148632
1148632
1148632
1148632
538347
538347
538347
538347
538347
538347
1165756
1165756
1165756
1165756
1165756
1165756
533576
533576
533576
533576
533576
533576
513303
513303
513303
513303
513303
513303
564471
1128942
1128942
564471
564471
1128942
550233
550233
1100466
550233
550233
550233
543544
543544
543544
543544
543544
543544
539658
539658
539658
539658
539658
539658
559072
559072
1118144
559072
559072
559072
544966
544966
544966
544966
544966
544966
532842
532842
532842
532842
532842
532842
544734
544734
544734
544734
544734
544734
540757
540757
540757
540757
540757
540757
542141
542141
542141
542141
542141
542141
530072
530072
530072
530072
530072
530072
538829
538829
538829
538829
538829
538829
541023
541023
541023
541023
541023
541023
555735
555735
555735
555735
555735
555735
1193444
1193444
1193444
1193444
1193444
1193444
571797
571797
1143594
1143594
571797
1143594
550118
550118
550118
550118
550118
550118
566764
566764
1133528
566764
566764
566764
555628
555628
555628
1111256
555628
555628
560112
560112
560112
560112
560112
560112
558385
558385
558385
1116770
558385
558385
540215
540215
540215
540215
540215
540215
550617
550617
550617
550617
550617
550617
since nobody reported a similar problem i guess the problem must be in my
code.... but i cannot imagine what i do wrong since the mapred job runs,
once configured, in the hadoop jobrunner.
"Devaraj Das" <[EMAIL PROTECTED]>
04/16/2008 10:43 AM
Please respond to
[email protected]
To
<[email protected]>
cc
Subject
RE: Counters giving double values
Input group refers to the number of <key, {set-of-values}> that the
reducer
gets. So if your map output looked like
<K, 1>
<K, 1>
To the reducer, the number of input groups would be 1 - <K, {1,1}> and the
number of records would be 2.
The fact that the number of reduce input records doubled beats me ...
Could
you please let us know the map output key/value types and if the types are
not the hadoop built-in types, pls also paste the code for
serializing/deserializing them.
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, April 16, 2008 12:41 PM
> To: [email protected]
> Subject: Re: Counters giving double values
>
> Hello,
>
> please provide me some help by giving me answers for the following
> questions:
>
> Is my assumption right, that the within a mapred job, the
> number of "Map output records" should/must match the number
> "Reduce input records" when no combiner is used in between?
>
> Can i screw the code in a way that the "Reduce input records"
> is twice the value of "Map output records"? And if, what
> could be my mistake?
>
> What is an "input group"?
>
> Any input is appreciated,
>
> ud
>
>
>
>
>
>
>
>
> [EMAIL PROTECTED]
> 04/10/2008 05:54 PM
> Please respond to
> [email protected]
>
>
> To
> [email protected]
> cc
>
> Subject
> Re: Counters giving double values
>
>
>
>
>
>
> Hello list readers,
>
> i'm still looking for an explanation.
>
> maybe i put it into this way: i run a job and at the end i
> need to know
> how many instances of a specific type were written into an
> output file.
> i wanted to rely on the counters. but maybe this is not a good idea
> anyway. or what is the outcome if speculative execution is used?
>
> how can i get the number of records contained in a MapFile -
> without the
> need to read it?
>
> thank you for some input,
>
> ud
>
>
>
>
>
>
>
> [EMAIL PROTECTED]
> 04/09/2008 10:35 AM
> Please respond to
> [email protected]
>
>
> To
> [email protected]
> cc
>
> Subject
> Counters giving double values
>
>
>
>
>
>
> hello,
>
> i tried to track down the problem with wrong counter values.
> didnt find
> any information / cases of it. maybe it is a feature i don't
> understand.
> the problem is, that in a local installation (localjobrunner)
> i sometimes
> (somehow randomly) get wrong counter values as the log()
> function of the
> Counters tells me in the log:
>
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Job
> complete: job_local_84
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Counters: 21
> 20080409044008268 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Map-Reduce Framework
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> input records=43917
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> output records=43917
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> input bytes=2116832
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0]
> JobClient - Map
> output bytes=9310354
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Combine input records=0
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Combine output records=0
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce input groups=83736
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce input records=87834
> 20080409044008269 INFO [DefaultQuartzScheduler_Worker-0] JobClient -
> Reduce output records=86640
>
> what i don't not understand is, why or how it comes, that the
> number of
> map output records is different than the reduce input records.
> maybe somebody has a simple explanation for this effect.
>
> i would expect that the number must be same. as you can see
> the number is
> (always) exactly the double value of what i expect it should be.
> and also when i check the output file, the amount of records
> matches the
> numbers from the map counters and not the double values.
>
> thanks,
>
> ud
>
>
>
>
>
>
>
1606742 1606742 1606742 803371 1606742 803371
770238 770238 770238 770238 770238 770238
743497 743497 743497 743497 743497 743497
788210 788210 788210 788210 788210 788210
765884 765884 765884 765884 765884 765884
784747 784747 784747 784747 784747 784747
760986 760986 760986 760986 760986 760986
746474 746474 746474 746474 746474 746474
742191 742191 742191 742191 742191 742191
716320 716320 716320 716320 716320 716320
696199 696199 696199 696199 696199 696199
732110 732110 732110 732110 732110 732110
739052 739052 739052 739052 739052 739052
756332 756332 756332 756332 756332 756332
744607 744607 744607 744607 744607 744607
748204 748204 748204 748204 748204 748204
736882 736882 736882 736882 736882 736882
700750 700750 700750 700750 700750 700750
259806 259806 259806 259806 259806 259806
1279252 639626 1279252 1279252 1279252 1279252
1270780 1270780 1270780 1270780 1270780 1270780
524832 524832 524832 524832 524832 524832
520642 520642 520642 520642 520642 520642
534722 534722 534722 534722 534722 534722
561428 1122856 1122856 561428 561428 1122856
535732 535732 535732 535732 535732 535732
525191 525191 525191 525191 525191 525191
515451 515451 515451 515451 515451 515451
531535 531535 531535 531535 531535 531535
502990 502990 502990 502990 502990 502990
515943 515943 515943 515943 515943 515943
523106 523106 523106 523106 523106 523106
510348 510348 510348 510348 510348 510348
535886 535886 535886 535886 535886 535886
522792 522792 522792 522792 522792 522792
535406 535406 535406 535406 535406 535406
523322 1046644 523322 523322 523322 523322
546711 546711 546711 546711 546711 546711
565917 565917 1131834 1131834 565917 1131834
564474 1128948 1128948 1128948 564474 1128948
1148090 1148090 1148090 1148090 1148090 1148090
565539 1131078 1131078 565539 565539 1131078
567117 1134234 1134234 567117 1134234 1134234
549599 549599 1099198 1099198 549599 549599
574316 1148632 1148632 1148632 1148632 1148632
538347 538347 538347 538347 538347 538347
1165756 1165756 1165756 1165756 1165756 1165756
533576 533576 533576 533576 533576 533576
513303 513303 513303 513303 513303 513303
564471 1128942 1128942 564471 564471 1128942
550233 550233 1100466 550233 550233 550233
543544 543544 543544 543544 543544 543544
539658 539658 539658 539658 539658 539658
559072 559072 1118144 559072 559072 559072
544966 544966 544966 544966 544966 544966
532842 532842 532842 532842 532842 532842
544734 544734 544734 544734 544734 544734
540757 540757 540757 540757 540757 540757
542141 542141 542141 542141 542141 542141
530072 530072 530072 530072 530072 530072
538829 538829 538829 538829 538829 538829
541023 541023 541023 541023 541023 541023
555735 555735 555735 555735 555735 555735
1193444 1193444 1193444 1193444 1193444 1193444
571797 571797 1143594 1143594 571797 1143594
550118 550118 550118 550118 550118 550118
566764 566764 1133528 566764 566764 566764
555628 555628 555628 1111256 555628 555628
560112 560112 560112 560112 560112 560112
558385 558385 558385 1116770 558385 558385
540215 540215 540215 540215 540215 540215
550617 550617 550617 550617 550617 550617