On Jan 15, 2008, at 22:09, Jim the Standing Bear wrote:
Only thing I noticed (compared to my code) is missing:
client.setConf(conf);
before client.run(conf).
In this case default input format is used which uses
LongWriteableComparable as
a key. Not sure if this is a case, but something worth checking.
Vadim
Well, I also wish it was this simple, but as I said in the original
message, I never wanted to use LongWritable at all. Here is how I set
the job conf, and after that, is the reduce task. Also, if I got the
incorrect output key/value type, shouldn't it always fail as soon as
the reduce task is run? But my code behaves strangely that sometimes
the exception didn't get thrown until a few iterations had been
successfully passed... Does the code reveal something that I missed?
Thanks.
JobConf countNewCatalogJobConf = new
JobConf(ThreddsCatalogIndexer.class);
countNewCatalogJobConf.setJobName("Count-New-Catalog-" +
iteration);
countNewCatalogJobConf.setInputPath(newCatUrlDir);
countNewCatalogJobConf.setInputFormat(KeyValueTextInputFormat.class);
countNewCatalogJobConf.setOutputPath(newCatalogCountDir);
countNewCatalogJobConf.setOutputKeyClass(Text.class);
countNewCatalogJobConf.setOutputValueClass(Text.class);
countNewCatalogJobConf.setReducerClass(NewCatalogCounter.class);
countNewCatalogJobConf.setNumReduceTasks(1);
JobClient.runJob(countNewCatalogJobConf);
public void reduce(WritableComparable key, Iterator values,
OutputCollector output, Reporter reporter) throws IOException {
long sum = 0;
if (key.toString().equals("NEWCAT")) {
while (values.hasNext()) {
sum++;
}
}
Text sumText = new Text();
String sumString = (new Long(sum)).toString();
sumText.set(sumString);
output.collect(key, sumText);
}
