I am sorry I overlooked something. Try to add the following:
job.setJarByClass(MyValueAggregatorJob.class); JobClient.runJob(job); To your main. Runping > -----Original Message----- > From: C G [mailto:[EMAIL PROTECTED] > Sent: Friday, October 12, 2007 8:05 PM > To: [email protected] > Subject: RE: Question about valueaggregators in 0.14.1... > > Hi Runping and All: > > I took your suggestion, renamed my original code to testAggregatorPlugin > and wrote a class testAggregator which contained the main as you show > here. It also fails with the same traceback...so it seems something is > profoundly wrong here. > > Is anybody else using the aggregation classes in 0.14.1 with success? > > This is ironic and frustrating...I have several other programs which use > map/reduce that are quite complex and those are running just fine. This > little plugin is all that stands in the way of delivering the system :-(. > > Any other comments/help, etc. most welcome... > > Thanks, > C G > > Runping Qi <[EMAIL PROTECTED]> wrote: > > Hadoop ignored the jar file you specified. > This is a known problem andhopefully HADOOP-1622 > (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem. > > In the meantime, you can create a class like below and add it to your jar: > > package your_package; > > > public class MyValueAggregatorJob{ > > public static void main(String args[]) throws IOException { > JobConf job = ValueAggregatorJob.createValueAggregatorJob(args); > JobClient.runJob(job); > } > } > > > And use the above class as the main class when you submit hadoop job: > bin/hadoop jar hadoop-0.14.1-dev-company.jar > your_package.MyValueAggregatorJob /input/set1 /output/set1 8 > textinputformat > testAggregator.xml > > > Hope this solve your problem. > > Runping > > > > > > -----Original Message----- > > From: C G [mailto:[EMAIL PROTECTED] > > Sent: Friday, October 12, 2007 2:43 PM > > To: [email protected] > > Subject: Question about valueaggregators in 0.14.1... > > > > Hi All: > > > > I am tearing out my hair trying to get a simple valueaggregator to run. > > This seems like an > > easy enough thing, but I am consistently getting hit with > > ClassNotFoundException which > > makes no sense to me. Any help would be immensely appreciated. Note > > that I've had > > valueaggregators running on 0.13.*. I'm a little concerned there is a > bug > > in 0.14.1. On the > > other hand, maybe I'm just blind to something obvious and just need a > > boot to the head... > > > > The environment is hadoop-0.14.1. I am running Java 1.6: > > $ java -version > > java version "1.6.0_02" > > Java(TM) SE Runtime Environment (build 1.6.0_02-b05) > > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode) > > > > The class name is called testAggregator, and exists in my jar file: > > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator > > 1826 Fri Oct 12 16:30:14 EDT 2007 > > com/company/hadoop/metrics/testAggregator.class > > > > My XML file is testAggregator.xml and looks like: > > > > > > > > > > > aggregator.descriptor.num > > 1 > > > > > > > > aggregator.descriptor.0 > > UserDefined,com.company.hadoop.metrics.testAggregator > > > > > > > > > My code couldn't be simpler: > > > > package com.company.hadoop.metrics; > > // snip a bunch of imports > > public class testAggregator extends ValueAggregatorBaseDescriptor { > > public ArrayList generateKeyValPairs(Object key, Object val) { > > String fields[] = val.toString().split("\t"); > > ArrayList retv = new ArrayList(); > > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE)); > > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text( > > fields[ 2 ]))); > > return retv; > > } > > public void configure(JobConf job) {} > > } > > > > I invoke things this way: > > > > bin/hadoop jar hadoop-0.14.1-dev-company.jar > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1 > > /output/set1 8 textinputformat testAggregator.xml > > > > All my jobs fail with a ClassNotFoundException like the following > > traceback: > > > > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > > Initializing JVM Metrics with processName=MAP, sessionId= > > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask: > > numReduceTasks: 8 > > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error > > running child > > java.lang.RuntimeException: java.lang.ClassNotFoundException: > > com.company.hadoop.metrics.testAggregator > > at > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > r.createInstance > > (UserDefinedValueAggregatorDescriptor.java:56) > > at > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63) > > at > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > r.(UserDefinedValueAggregatorDescriptor.java:75) > > at > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr > > egatorDescriptor(ValueAggregatorJobBase.java:49) > > at > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato > > rDescriptors(ValueAggregatorJobBase.java:60) > > at > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy > > Spec(ValueAggregatorJobBase.java:69) > > at > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va > > lueAggregatorJobBase.java:37) > > at > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > > at > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82 > > ) > > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32) > > at > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > > at > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82 > > ) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185) > > at > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777) > > Caused by: java.lang.ClassNotFoundException: > > com.company.hadoop.metrics.testAggregator > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > > at java.lang.Class.forName0(Native Method) > > at java.lang.Class.forName(Class.java:247) > > at > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > r.createInstance > > (UserDefinedValueAggregatorDescriptor.java:51) > > ... 13 more > > > > As near as I can tell I am doing everything correctly. Other code > > existing in this jar file is > > running just fine - in fact it was used to produce the input to this > > test. I've checked my > > systems to make I've got the jar file (and XML file) onto each of the 4 > > compute nodes in the > > system. > > > > Can anybody spot my problem or tell me what to do to get this going? > > > > Thanks, > > C G > > > > > > --------------------------------- > > Tonight's top picks. What will you watch tonight? Preview the hottest > > shows on Yahoo! TV. > > > > > --------------------------------- > Be a better Heartthrob. Get better relationship answers from someone who > knows. > Yahoo! Answers - Check it out.
