Thanks! Will take a look at the jira issue 3267

  _____  

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] 
Sent: Thursday, April 17, 2008 7:09 PM
To: [email protected]
Subject: RE: Counters giving double values



hi devraj, 

so, i researched the topic with the counters further with some success. 
for one i can reproduce it now with a Test. 

i am waiting for the password for my JIRA account to get started there -
somehow i didnt get the password after registration, i sent a mail to owen. 
i am not familiar with the proceedings on JIRA / ASF... so if you see
mistakes please guide me. thanks. 

if you are interested, try out the attached testcase. i figured it is some
timing issue within the localjobrunner (in my case, dont know about
distributed running yet)  and i will try to provide a patch. 
can i now submit this testcase trough JIRA? do i need to change something on
it? 

kind regards, 

ud 







"Devaraj Das" <[EMAIL PROTECTED]> 


04/16/2008 01:18 PM 


Please respond to
[email protected]



To
<[email protected]> 

cc

Subject
RE: Counters giving double values

        




Pls file a jira for the counter updates part. It will be excellent if you
can also attach a testcase that can reproduce the problem (maybe a stripped
down version of your app or something). 

> -----Original Message-----
> From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL PROTECTED] 
> Sent: Wednesday, April 16, 2008 4:35 PM
> To: [email protected]
> Subject: RE: Counters giving double values
> 
> hadoop 0.16.2
> (and as i remember, i had the same issue with 0.16.0)
> 
> Yes, the final data output at the end IS CORRECT. 
> only the counter values are wrong.
> 
> i didnt try to run it in a distributed environment yet. only local.
> 
> 
> 
> 
> 
> 
> "Devaraj Das" <[EMAIL PROTECTED]>
> 04/16/2008 12:56 PM
> Please respond to
> [email protected]
> 
> 
> To
> <[email protected]>
> cc
> 
> Subject
> RE: Counters giving double values
> 
> 
> 
> 
> 
> 
> Also, in those cases where you see wrong counter values, did 
> you validate
> the final (reduce) output for correctness (I am just trying 
> to see whether
> the problem is with the Counter updates). 
> 
> > -----Original Message-----
> > From: Devaraj Das [ <mailto:[EMAIL PROTECTED]>
mailto:[EMAIL PROTECTED] 
> > Sent: Wednesday, April 16, 2008 4:23 PM
> > To: [email protected]
> > Subject: RE: Counters giving double values
> > 
> > Thanks for the detailed answer. Which hadoop version are you 
> > on? If you are confident that it is not a problem with your 
> > app, pls raise a jira.
> > 
> > 
> >   _____ 
> > 
> > From: [EMAIL PROTECTED] [ <mailto:[EMAIL PROTECTED]> mailto:[EMAIL 
> > PROTECTED]
> > Sent: Wednesday, April 16, 2008 3:25 PM
> > To: [email protected]
> > Subject: RE: Counters giving double values
> > 
> > 
> > 
> > Thanks so far. 
> > 
> > key and values are custom implementations. 
> > 
> > key implements WritableComparable 
> > value extends VersionedWritable 
> > 
> > btw. The only problem i encounter is that the Counter values 
> > are wrong. If i
> > check the records in the MapFile (re-read it) which is 
> > written as the output
> > of the mapred job, the amount of records is correct and 
> > represents the halve
> > of the reported counter value. 
> > the same applies for the results of the operations which are 
> > carried out in
> > the reduce(). everything is correct, except the counter values. 
> > 
> > the whole thing happens only sometimes. 
> > 
> > 
> > Key serializing / Deserializing, i guess you want to see this 
> > part of the
> > code: 
> > 
> >         public int language; 
> >         public String term; 
> > 
> >         public void readFields(DataInput in) throws IOException { 
> >                 language = in.readInt(); 
> >                 term =         Text.readString(in).toString(); 
> >         } 
> > 
> >         public void write(DataOutput out) throws IOException { 
> >                 out.writeInt(language); 
> >                 Text.writeString(out, term); 
> >         } 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 




Reply via email to