Re: output multiple values?

[Shirley Cohen]

Thanks Owen! I found the bug in my code: Doing collect twice does  
work now :))


Shirley

On Sep 9, 2008, at 4:19 PM, Owen O'Malley wrote:



On Sep 9, 2008, at 12:20 PM, Shirley Cohen wrote:

I have a simple reducer that computes the average by doing a sum/ 
count. But I want to output both the average and the count for a  
given key, not just the average. Is it possible to output both  
values from the same invocation of the reducer? Or do I need two  
reducer invocations? If I try to call output.collect() twice from  
the reducer and label the key with "type=avg" or "type=count", I  
get a bunch of garbage out. Please let me know if you have any  
suggestions.


I'd be tempted to define a type like:

class AverageAndCount implements Writable {
  private long sum;
  private long count;
  ...
  public String toString() {
 return "avg = " + (sum / (double) count) + ", count = " + count);
  }
}

Then you could use your reducer as both a combiner and reducer and  
you would get both values out if you use TextOutputFormat. That  
said, it should absolutely work to do collect twice.


-- Owen

Re: output multiple values?

[Owen O'Malley]

On Sep 9, 2008, at 12:20 PM, Shirley Cohen wrote:

I have a simple reducer that computes the average by doing a sum/ 
count. But I want to output both the average and the count for a  
given key, not just the average. Is it possible to output both  
values from the same invocation of the reducer? Or do I need two  
reducer invocations? If I try to call output.collect() twice from  
the reducer and label the key with "type=avg" or "type=count", I get  
a bunch of garbage out. Please let me know if you have any  
suggestions.


I'd be tempted to define a type like:

class AverageAndCount implements Writable {
  private long sum;
  private long count;
  ...
  public String toString() {
 return "avg = " + (sum / (double) count) + ", count = " + count);
  }
}

Then you could use your reducer as both a combiner and reducer and you  
would get both values out if you use TextOutputFormat. That said, it  
should absolutely work to do collect twice.


-- Owen

output multiple values?

[Shirley Cohen]

I have a simple reducer that computes the average by doing a sum/ 
count. But I want to output both the average and the count for a  
given key, not just the average. Is it possible to output both values  
from the same invocation of the reducer? Or do I need two reducer  
invocations? If I try to call output.collect() twice from the reducer  
and label the key with "type=avg" or "type=count", I get a bunch of  
garbage out. Please let me know if you have any suggestions.


Thanks,

Shirley