Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
In the meantime, you can create a class like below and add it to your jar:
package your_package;
public class MyValueAggregatorJob{
public static void main(String args[]) throws IOException {
JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
JobClient.runJob(job);
}
}
And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml
Hope this solve your problem.
Runping
> -----Original Message-----
> From: C G [mailto:[EMAIL PROTECTED]
> Sent: Friday, October 12, 2007 2:43 PM
> To: [email protected]
> Subject: Question about valueaggregators in 0.14.1...
>
> Hi All:
>
> I am tearing out my hair trying to get a simple valueaggregator to run.
> This seems like an
> easy enough thing, but I am consistently getting hit with
> ClassNotFoundException which
> makes no sense to me. Any help would be immensely appreciated. Note
> that I've had
> valueaggregators running on 0.13.*. I'm a little concerned there is a bug
> in 0.14.1. On the
> other hand, maybe I'm just blind to something obvious and just need a
> boot to the head...
>
> The environment is hadoop-0.14.1. I am running Java 1.6:
> $ java -version
> java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
>
> The class name is called testAggregator, and exists in my jar file:
> $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> 1826 Fri Oct 12 16:30:14 EDT 2007
> com/company/hadoop/metrics/testAggregator.class
>
> My XML file is testAggregator.xml and looks like:
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
> <property>
> <name>aggregator.descriptor.num</name>
> <value>1</value>
> </property>
> <property>
> <name>aggregator.descriptor.0</name>
> <value>UserDefined,com.company.hadoop.metrics.testAggregator</value>
> </property>
> </configuration>
>
> My code couldn't be simpler:
>
> package com.company.hadoop.metrics;
> // snip a bunch of imports
> public class testAggregator extends ValueAggregatorBaseDescriptor {
> public ArrayList<Entry> generateKeyValPairs(Object key, Object val) {
> String fields[] = val.toString().split("\t");
> ArrayList<Entry> retv = new ArrayList<Entry>();
> retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> fields[ 2 ])));
> return retv;
> }
> public void configure(JobConf job) {}
> }
>
> I invoke things this way:
>
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> /output/set1 8 textinputformat testAggregator.xml
>
> All my jobs fail with a ClassNotFoundException like the following
> traceback:
>
> 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 8
> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:56)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.<init>(UserDefinedValueAggregatorDescriptor.java:75)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> egatorDescriptor(ValueAggregatorJobBase.java:49)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> rDescriptors(ValueAggregatorJobBase.java:60)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> Spec(ValueAggregatorJobBase.java:69)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> lueAggregatorJobBase.java:37)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> Caused by: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
>
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:51)
> ... 13 more
>
> As near as I can tell I am doing everything correctly. Other code
> existing in this jar file is
> running just fine - in fact it was used to produce the input to this
> test. I've checked my
> systems to make I've got the jar file (and XML file) onto each of the 4
> compute nodes in the
> system.
>
> Can anybody spot my problem or tell me what to do to get this going?
>
> Thanks,
> C G
>
>
> ---------------------------------
> Tonight's top picks. What will you watch tonight? Preview the hottest
> shows on Yahoo! TV.
