After a little side-bar email, Ted suggested the following: ------- "I suspect that the construction of conf is insufficient to let one of your jobs know about which jar file to use (likely the Canopy clustering). This is a confusing aspect of hadoop.
The problem (as I think I see it) is that when you run the clustering job, the clustering code instantiates a configuration and runs a job from that. The actual job jar is constructed based on hints in that configuration. That will cause the canopy clustering stuff to be included, but not your jar. The fix is that you need to have the Canopy clustering code include a reference back to where it finds the distance measure. ..." ------- I modified the jar argument to the CanopyClusteringJob to name my jar instead of the mahout jar and it worked! Now I finally understand why Grant added that jar parameter to the canopy job. Posting this happy result to the list so others can benefit from it in the future. Thanks Ted, (Grant), Jeff -----Original Message----- From: Jeff Eastman [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 05, 2008 11:53 AM To: [email protected] Subject: RE: Class Loader Problem Here's my job driver, it works fine with ManhattanDistanceMeasure but not SystemLoadDistanceMeasure. Jeff public static void main(String[] args) { String input = args[0]; String output = args[1]; int t1 = new Integer(args[2]); int t2 = new Integer(args[3]); JobConf conf = new JobConf( com.collabnet.hadoop.systemload.access.DriverA.class); Path outPath = new Path(output); try { FileSystem dfs = FileSystem.get(conf); if (dfs.exists(outPath)) dfs.delete(outPath); DriverA.runJob(input, output); DriverP.runJob(input, output); DriverC.runJob(output, output); CanopyClusteringJob.runJob(output + "/combined", output, SystemLoadDistanceMeasure.class.getName(), t1, t2, "apache-mahout-0.1-dev.jar"); DriverS.runJob(output + "/clusters", output); } catch (IOException e) { e.printStackTrace(); } } -----Original Message----- From: Ted Dunning [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 05, 2008 11:44 AM To: [email protected] Subject: Re: Class Loader Problem Where is your code? On 3/5/08 11:28 AM, "Jeff Eastman" <[EMAIL PROTECTED]> wrote: > I'm wondering if you can see anything > wrong with my packaging or, perhaps, how the Canopy class is going about > instantiating it.
