Glad it works. HADOOP-1622 should fix the problem properly. Until then, the users have to use this kind of hacky obscure workaround:)
Runping > -----Original Message----- > From: C G [mailto:[EMAIL PROTECTED] > Sent: Friday, October 12, 2007 8:48 PM > To: [email protected] > Subject: RE: Question about valueaggregators in 0.14.1... > > Hi Runping and All: > > That fixed the problem. Of course my aggregator is now failing for a > different reason, but that's an error in my code that I can fix. > > I am extremely grateful for your assistance! > > Thanks, > C G > > Runping Qi <[EMAIL PROTECTED]> wrote: > > > I am sorry I overlooked something. > Try to add the following: > > job.setJarByClass(MyValueAggregatorJob.class); > JobClient.runJob(job); > > To your main. > > Runping > > > -----Original Message----- > > From: C G [mailto:[EMAIL PROTECTED] > > Sent: Friday, October 12, 2007 8:05 PM > > To: [email protected] > > Subject: RE: Question about valueaggregators in 0.14.1... > > > > Hi Runping and All: > > > > I took your suggestion, renamed my original code to testAggregatorPlugin > > and wrote a class testAggregator which contained the main as you show > > here. It also fails with the same traceback...so it seems something is > > profoundly wrong here. > > > > Is anybody else using the aggregation classes in 0.14.1 with success? > > > > This is ironic and frustrating...I have several other programs which use > > map/reduce that are quite complex and those are running just fine. This > > little plugin is all that stands in the way of delivering the system :- > (. > > > > Any other comments/help, etc. most welcome... > > > > Thanks, > > C G > > > > Runping Qi wrote: > > > > Hadoop ignored the jar file you specified. > > This is a known problem andhopefully HADOOP-1622 > > (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the > problem. > > > > In the meantime, you can create a class like below and add it to your > jar: > > > > package your_package; > > > > > > public class MyValueAggregatorJob{ > > > > public static void main(String args[]) throws IOException { > > JobConf job = ValueAggregatorJob.createValueAggregatorJob(args); > > JobClient.runJob(job); > > } > > } > > > > > > And use the above class as the main class when you submit hadoop job: > > bin/hadoop jar hadoop-0.14.1-dev-company.jar > > your_package.MyValueAggregatorJob /input/set1 /output/set1 8 > > textinputformat > > testAggregator.xml > > > > > > Hope this solve your problem. > > > > Runping > > > > > > > > > > > -----Original Message----- > > > From: C G [mailto:[EMAIL PROTECTED] > > > Sent: Friday, October 12, 2007 2:43 PM > > > To: [email protected] > > > Subject: Question about valueaggregators in 0.14.1... > > > > > > Hi All: > > > > > > I am tearing out my hair trying to get a simple valueaggregator to > run. > > > This seems like an > > > easy enough thing, but I am consistently getting hit with > > > ClassNotFoundException which > > > makes no sense to me. Any help would be immensely appreciated. Note > > > that I've had > > > valueaggregators running on 0.13.*. I'm a little concerned there is a > > bug > > > in 0.14.1. On the > > > other hand, maybe I'm just blind to something obvious and just need a > > > boot to the head... > > > > > > The environment is hadoop-0.14.1. I am running Java 1.6: > > > $ java -version > > > java version "1.6.0_02" > > > Java(TM) SE Runtime Environment (build 1.6.0_02-b05) > > > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode) > > > > > > The class name is called testAggregator, and exists in my jar file: > > > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator > > > 1826 Fri Oct 12 16:30:14 EDT 2007 > > > com/company/hadoop/metrics/testAggregator.class > > > > > > My XML file is testAggregator.xml and looks like: > > > > > > > > > > > > > > > > > aggregator.descriptor.num > > > 1 > > > > > > > > > > > > > aggregator.descriptor.0 > > > UserDefined,com.company.hadoop.metrics.testAggregator > > > > > > > > > > > > > > My code couldn't be simpler: > > > > > > package com.company.hadoop.metrics; > > > // snip a bunch of imports > > > public class testAggregator extends ValueAggregatorBaseDescriptor { > > > public ArrayList generateKeyValPairs(Object key, Object val) { > > > String fields[] = val.toString().split("\t"); > > > ArrayList retv = new ArrayList(); > > > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE)); > > > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text( > > > fields[ 2 ]))); > > > return retv; > > > } > > > public void configure(JobConf job) {} > > > } > > > > > > I invoke things this way: > > > > > > bin/hadoop jar hadoop-0.14.1-dev-company.jar > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1 > > > /output/set1 8 textinputformat testAggregator.xml > > > > > > All my jobs fail with a ClassNotFoundException like the following > > > traceback: > > > > > > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > > > Initializing JVM Metrics with processName=MAP, sessionId= > > > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask: > > > numReduceTasks: 8 > > > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: > Error > > > running child > > > java.lang.RuntimeException: java.lang.ClassNotFoundException: > > > com.company.hadoop.metrics.testAggregator > > > at > > > > > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > > r.createInstance > > > (UserDefinedValueAggregatorDescriptor.java:56) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > > r.(UserDefinedValueAggregatorDescriptor.java:75) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr > > > egatorDescriptor(ValueAggregatorJobBase.java:49) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato > > > rDescriptors(ValueAggregatorJobBase.java:60) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy > > > Spec(ValueAggregatorJobBase.java:69) > > > at > > > > > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va > > > lueAggregatorJobBase.java:37) > > > at > > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > > > at > > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82 > > > ) > > > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32) > > > at > > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58) > > > at > > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82 > > > ) > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185) > > > at > > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777) > > > Caused by: java.lang.ClassNotFoundException: > > > com.company.hadoop.metrics.testAggregator > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:251) > > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) > > > at java.lang.Class.forName0(Native Method) > > > at java.lang.Class.forName(Class.java:247) > > > at > > > > > > > > > org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto > > > r.createInstance > > > (UserDefinedValueAggregatorDescriptor.java:51) > > > ... 13 more > > > > > > As near as I can tell I am doing everything correctly. Other code > > > existing in this jar file is > > > running just fine - in fact it was used to produce the input to this > > > test. I've checked my > > > systems to make I've got the jar file (and XML file) onto each of the > 4 > > > compute nodes in the > > > system. > > > > > > Can anybody spot my problem or tell me what to do to get this going? > > > > > > Thanks, > > > C G > > > > > > > > > --------------------------------- > > > Tonight's top picks. What will you watch tonight? Preview the hottest > > > shows on Yahoo! TV. > > > > > > > > > > --------------------------------- > > Be a better Heartthrob. Get better relationship answers from someone who > > knows. > > Yahoo! Answers - Check it out. > > > > > --------------------------------- > Catch up on fall's hot new shows on Yahoo! TV. Watch previews, get > listings, and more!
