Simple SetWritable class

[CloudyEye]

I tried to extend TreeSet to be Wrtiable. Here is what I did.

public class SetWritable extends TreeSet implements
WritableComparable {

public void readFields(DataInput in) throws IOException {
clear();
int sz=in.readInt();
for (int i = 0; i < sz; i++) {
add(in.readInt());
}
}

public void write(DataOutput out) throws IOException {
out.writeInt(size());
Iterator iter=this.iterator();
while (iter.hasNext()) {
out.writeInt(iter.next());
}
}   
}


 If I remove clear() from the readFields() I am gettting wrong output.(some
old data is written with the new ones !). With "clear()" it is ok as my
simple tests show.

Is this implementaion save to be used ?

Regards,



-- 
View this message in context: 
http://www.nabble.com/Simple-SetWritable-class-tp16833976p16833976.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Using ArrayWritable of type IntWritable

[CloudyEye]

Doug Cutting-4 wrote:
> 
> public String toString();
> 
> Doug
> 

That's it.
 Many thanks.
-- 
View this message in context: 
http://www.nabble.com/Using-ArrayWritable-of-type-IntWritable-tp16807489p16823104.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Using ArrayWritable of type IntWritable

[CloudyEye]

Hi,

>From the API , I should create new class as follows:

public class IntArrayWritable extends ArrayWritable { 
public IntArrayWritable() { 
super(IntWritable.class); 
}
 }

In the reducer, When executing 
OutputCollector.collect(WritableComparable key, IntArrayWritable arr)

I am getting something like:
key1[EMAIL PROTECTED]
key2[EMAIL PROTECTED]
key3[EMAIL PROTECTED]
key4[EMAIL PROTECTED]
...
...

What else do I have to override in "ArrayWritable " to get the IntWritable
values written to the output files by the reducers?
If you have some ready code for such case I will be very thankful.

Cheers,

-- 
View this message in context: 
http://www.nabble.com/Using-ArrayWritable-of-type-IntWritable-tp16807489p16807489.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[CloudyEye]

Thank you Mr. Udatny. Your code was helpful.

Cheers,

-- 
View this message in context: 
http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16722296.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[CloudyEye]

Thank you. I tried a while with some code but I did not succeed in getting
the value of counters. Please check this code:

public class MyMapper extends MapReduceBase implements Mapper {
private static enum myCounters {NUMBER_OF_ROWS}

public void map(WritableComparable key, Writable values,
OutputCollector output, Reporter reporter) throws 
IOException {

reporter.incrCounter(myCounters.NUMBER_OF_ROWS, 1); 
}
}

And here is my simple reducer:
public class MyReduce extends MapReduceBase implements Reducer {
private static enum myCounters {NUMBER_OF_ROWS}
private static long numOfRows;

public void configure(JobConf job) {
super.configure(job);

this.numOfRows= 

}
}

How can I assign the value of counter: NUMBER_OF_ROWS to the long variable
numOfRows  ??

Best regards,




stack-3 wrote:
> 
> https://issues.apache.org/jira/browse/HBASE-559 has an example. Ignore 
> the HBase stuff.  Whats important is the ENUM at head of the MR job 
> class, the calls to Reporter inside in tasks, and the properties file -- 
> both how its named and that it ends up in the generated job jar.
> St.Ack
> 
> 

-- 
View this message in context: 
http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16719201.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

How can I use counters in Hadoop

[CloudyEye]

Hi, I am new newbie to Hadoop. I would be thankful if you help me.

I've read that I can use the Reporter class to increase counters. this way:
reporter.incrCounter(Enum args, long arg1);

How can I get the values of those counters ?

My aim is to count the total inputs to the mappers , then i want to override
the:
public void configure(JobConf job) {} method of the reducer to get the
counter value before reducing any key,values.

Regards,
-- 
View this message in context: 
http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16702607.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.