Hi All:
I am tearing out my hair trying to get a simple valueaggregator to run. This
seems like an
easy enough thing, but I am consistently getting hit with
ClassNotFoundException which
makes no sense to me. Any help would be immensely appreciated. Note that
I've had
valueaggregators running on 0.13.*. I'm a little concerned there is a bug in
0.14.1. On the
other hand, maybe I'm just blind to something obvious and just need a boot
to the head...
The environment is hadoop-0.14.1. I am running Java 1.6:
$ java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
The class name is called testAggregator, and exists in my jar file:
$ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
1826 Fri Oct 12 16:30:14 EDT 2007
com/company/hadoop/metrics/testAggregator.class
My XML file is testAggregator.xml and looks like:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>aggregator.descriptor.num</name>
<value>1</value>
</property>
<property>
<name>aggregator.descriptor.0</name>
<value>UserDefined,com.company.hadoop.metrics.testAggregator</value>
</property>
</configuration>
My code couldn't be simpler:
package com.company.hadoop.metrics;
// snip a bunch of imports
public class testAggregator extends ValueAggregatorBaseDescriptor {
public ArrayList<Entry> generateKeyValPairs(Object key, Object val) {
String fields[] = val.toString().split("\t");
ArrayList<Entry> retv = new ArrayList<Entry>();
retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text( fields[ 2
])));
return retv;
}
public void configure(JobConf job) {}
}
I invoke things this way:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
/output/set1 8 textinputformat testAggregator.xml
All my jobs fail with a ClassNotFoundException like the following traceback:
2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 8
2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.company.hadoop.metrics.testAggregator
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance
(UserDefinedValueAggregatorDescriptor.java:56)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.<init>(UserDefinedValueAggregatorDescriptor.java:75)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggregatorDescriptor(ValueAggregatorJobBase.java:49)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregatorDescriptors(ValueAggregatorJobBase.java:60)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMySpec(ValueAggregatorJobBase.java:69)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(ValueAggregatorJobBase.java:37)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
Caused by: java.lang.ClassNotFoundException:
com.company.hadoop.metrics.testAggregator
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance
(UserDefinedValueAggregatorDescriptor.java:51)
... 13 more
As near as I can tell I am doing everything correctly. Other code existing
in this jar file is
running just fine - in fact it was used to produce the input to this test.
I've checked my
systems to make I've got the jar file (and XML file) onto each of the 4
compute nodes in the
system.
Can anybody spot my problem or tell me what to do to get this going?
Thanks,
C G
---------------------------------
Tonight's top picks. What will you watch tonight? Preview the hottest shows on
Yahoo! TV.
