oops...may have spotted the problem. I was accidentally looking at the *mapred* TotalOrderPartitioner source code. Looks as though the hadoopbackport TotalOrderPartitioner source is using the name "mapreduce.totalorderpartitioner.path" as the job configuration parameter name.
-----Original Message----- From: Geoff Hendrey [mailto:[email protected]] Sent: Wednesday, March 02, 2011 9:08 PM To: [email protected] Subject: follow-up question on TotalOrderPartitioner I upgraded my client to 0.90.1 per the suggestion (although the server is still 0.89). I no longer get a NullPointerException when I try to use TotalOrderPartitioner. However, I cannot get the TotalOrderPartitioner to actually create the partition file, even though a message is printed "hadoopbackport.InputSampler: Using 64 samples" which indicated my custom sampler is running and generating partition points. Can someone take a quick grock at my code below and tell me what I'm missing? I use a fully qualified path name, I even set "total.order.partitioner.path". All my println statements indicate the partitions file is created, but it isn't created. Even more strangely, when the mapreduce job starts, it complains that "File _partition.lst does not exist" (even though I've explicitly told it to use a file named "partitions-file" as opposed to the default "_partition.lst"). Path input = FileInputFormat.getInputPaths(job)[0]; input = input.makeQualified(input.getFileSystem(config)); Path partitionFilePath = new Path(input, "partitions-file"); TotalOrderPartitioner.setPartitionFile(config, partitionFilePath); job.getConfiguration().set("total.order.partitioner.path", partitionFilePath.toString()); job.setPartitionerClass(TotalOrderPartitioner.class); System.out.println("TotalOrderPartitioner thinks it's partition file is: " + TotalOrderPartitioner.getPartitionFile(config)); job.setNumReduceTasks(100); InputSampler.Sampler randomSampler = new RandomKeySampler<Text, HitList>(100); InputSampler.writePartitionFile(job, randomSampler); System.out.println("wrote partition file: " + TotalOrderPartitioner.getPartitionFile(config)); Any help greatly appreciated since I spent the day looking through TotalOrderPartitioner and can't find what I'm doing wrong. Thanks! -geoff
