Is there some magic to get a Partitioner working on .20.0? Setting
the partitioner class on the Job object doesn't take, hadoop always
uses the HashPartitioner. Looking through the source code, it looks
like the MapOutputBuffer in MapTask only ever fetches the
"mapred.partitioner.class", and doesn't check for new api's
"mapreduce.partitioner.class", but I'm not confident in my
understanding of how things work.
I was eventually able to get my test program working correctly by:
1) Creating a partitioner that extends the deprecated
org.apache.hadoop.mapred.Partitioner class.
2) Calling job.getConfiguration().set("mapred.partitioner.class",
DeprecatedTestPartitioner.class.getCanonicalName());
3) Commenting out line 395 of org.apache.hadoop.mapreduce.Job.java,
where it asserts that "mapred.partitioner.class" is null
But I'm assuming editing the hadoop core sourcecode is not the
intended path. Am I missing some simple switch or something?
rf
