Thanks! Will take a look at the jira issue 3267
_____ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, April 17, 2008 7:09 PM To: [email protected] Subject: RE: Counters giving double values hi devraj, so, i researched the topic with the counters further with some success. for one i can reproduce it now with a Test. i am waiting for the password for my JIRA account to get started there - somehow i didnt get the password after registration, i sent a mail to owen. i am not familiar with the proceedings on JIRA / ASF... so if you see mistakes please guide me. thanks. if you are interested, try out the attached testcase. i figured it is some timing issue within the localjobrunner (in my case, dont know about distributed running yet) and i will try to provide a patch. can i now submit this testcase trough JIRA? do i need to change something on it? kind regards, ud "Devaraj Das" <[EMAIL PROTECTED]> 04/16/2008 01:18 PM Please respond to [email protected] To <[email protected]> cc Subject RE: Counters giving double values Pls file a jira for the counter updates part. It will be excellent if you can also attach a testcase that can reproduce the problem (maybe a stripped down version of your app or something). > -----Original Message----- > From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] > Sent: Wednesday, April 16, 2008 4:35 PM > To: [email protected] > Subject: RE: Counters giving double values > > hadoop 0.16.2 > (and as i remember, i had the same issue with 0.16.0) > > Yes, the final data output at the end IS CORRECT. > only the counter values are wrong. > > i didnt try to run it in a distributed environment yet. only local. > > > > > > > "Devaraj Das" <[EMAIL PROTECTED]> > 04/16/2008 12:56 PM > Please respond to > [email protected] > > > To > <[email protected]> > cc > > Subject > RE: Counters giving double values > > > > > > > Also, in those cases where you see wrong counter values, did > you validate > the final (reduce) output for correctness (I am just trying > to see whether > the problem is with the Counter updates). > > > -----Original Message----- > > From: Devaraj Das [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] > > Sent: Wednesday, April 16, 2008 4:23 PM > > To: [email protected] > > Subject: RE: Counters giving double values > > > > Thanks for the detailed answer. Which hadoop version are you > > on? If you are confident that it is not a problem with your > > app, pls raise a jira. > > > > > > _____ > > > > From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL > > PROTECTED] > > Sent: Wednesday, April 16, 2008 3:25 PM > > To: [email protected] > > Subject: RE: Counters giving double values > > > > > > > > Thanks so far. > > > > key and values are custom implementations. > > > > key implements WritableComparable > > value extends VersionedWritable > > > > btw. The only problem i encounter is that the Counter values > > are wrong. If i > > check the records in the MapFile (re-read it) which is > > written as the output > > of the mapred job, the amount of records is correct and > > represents the halve > > of the reported counter value. > > the same applies for the results of the operations which are > > carried out in > > the reduce(). everything is correct, except the counter values. > > > > the whole thing happens only sometimes. > > > > > > Key serializing / Deserializing, i guess you want to see this > > part of the > > code: > > > > public int language; > > public String term; > > > > public void readFields(DataInput in) throws IOException { > > language = in.readInt(); > > term = Text.readString(in).toString(); > > } > > > > public void write(DataOutput out) throws IOException { > > out.writeInt(language); > > Text.writeString(out, term); > > } > > > > > > > > > > > > >
