Pls file a jira for the counter updates part. It will be excellent if you can also attach a testcase that can reproduce the problem (maybe a stripped down version of your app or something).
> -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > Sent: Wednesday, April 16, 2008 4:35 PM > To: [email protected] > Subject: RE: Counters giving double values > > hadoop 0.16.2 > (and as i remember, i had the same issue with 0.16.0) > > Yes, the final data output at the end IS CORRECT. > only the counter values are wrong. > > i didnt try to run it in a distributed environment yet. only local. > > > > > > > "Devaraj Das" <[EMAIL PROTECTED]> > 04/16/2008 12:56 PM > Please respond to > [email protected] > > > To > <[email protected]> > cc > > Subject > RE: Counters giving double values > > > > > > > Also, in those cases where you see wrong counter values, did > you validate > the final (reduce) output for correctness (I am just trying > to see whether > the problem is with the Counter updates). > > > -----Original Message----- > > From: Devaraj Das [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, April 16, 2008 4:23 PM > > To: [email protected] > > Subject: RE: Counters giving double values > > > > Thanks for the detailed answer. Which hadoop version are you > > on? If you are confident that it is not a problem with your > > app, pls raise a jira. > > > > > > _____ > > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > > Sent: Wednesday, April 16, 2008 3:25 PM > > To: [email protected] > > Subject: RE: Counters giving double values > > > > > > > > Thanks so far. > > > > key and values are custom implementations. > > > > key implements WritableComparable > > value extends VersionedWritable > > > > btw. The only problem i encounter is that the Counter values > > are wrong. If i > > check the records in the MapFile (re-read it) which is > > written as the output > > of the mapred job, the amount of records is correct and > > represents the halve > > of the reported counter value. > > the same applies for the results of the operations which are > > carried out in > > the reduce(). everything is correct, except the counter values. > > > > the whole thing happens only sometimes. > > > > > > Key serializing / Deserializing, i guess you want to see this > > part of the > > code: > > > > public int language; > > public String term; > > > > public void readFields(DataInput in) throws IOException { > > language = in.readInt(); > > term = Text.readString(in).toString(); > > } > > > > public void write(DataOutput out) throws IOException { > > out.writeInt(language); > > Text.writeString(out, term); > > } > > > > > > > > > > > > >
