HADOOP-1622 has a new patch available. That may solve your problem.
Dennis Kubes
C G wrote:
Hi Runping and All:
I took your suggestion, renamed my original code to testAggregatorPlugin and wrote a class testAggregator which contained the main as you show here. It also fails with the same traceback...so it seems something is profoundly wrong here.
Is anybody else using the aggregation classes in 0.14.1 with success?
This is ironic and frustrating...I have several other programs which use map/reduce that are quite complex and those are running just fine. This little plugin is all that stands in the way of delivering the system :-(.
Any other comments/help, etc. most welcome...
Thanks,
C G
Runping Qi <[EMAIL PROTECTED]> wrote:
Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
In the meantime, you can create a class like below and add it to your jar:
package your_package;
public class MyValueAggregatorJob{
public static void main(String args[]) throws IOException {
JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
JobClient.runJob(job);
}
}
And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml
Hope this solve your problem.
Runping
-----Original Message-----
From: C G [mailto:[EMAIL PROTECTED]
Sent: Friday, October 12, 2007 2:43 PM
To: [email protected]
Subject: Question about valueaggregators in 0.14.1...
Hi All:
I am tearing out my hair trying to get a simple valueaggregator to run.
This seems like an
easy enough thing, but I am consistently getting hit with
ClassNotFoundException which
makes no sense to me. Any help would be immensely appreciated. Note
that I've had
valueaggregators running on 0.13.*. I'm a little concerned there is a bug
in 0.14.1. On the
other hand, maybe I'm just blind to something obvious and just need a
boot to the head...
The environment is hadoop-0.14.1. I am running Java 1.6:
$ java -version
java version "1.6.0_02"
Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
The class name is called testAggregator, and exists in my jar file:
$ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
1826 Fri Oct 12 16:30:14 EDT 2007
com/company/hadoop/metrics/testAggregator.class
My XML file is testAggregator.xml and looks like:
aggregator.descriptor.num
1
aggregator.descriptor.0
UserDefined,com.company.hadoop.metrics.testAggregator
My code couldn't be simpler:
package com.company.hadoop.metrics;
// snip a bunch of imports
public class testAggregator extends ValueAggregatorBaseDescriptor {
public ArrayList generateKeyValPairs(Object key, Object val) {
String fields[] = val.toString().split("\t");
ArrayList retv = new ArrayList();
retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
fields[ 2 ])));
return retv;
}
public void configure(JobConf job) {}
}
I invoke things this way:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
/output/set1 8 textinputformat testAggregator.xml
All my jobs fail with a ClassNotFoundException like the following
traceback:
2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 8
2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
running child
java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.company.hadoop.metrics.testAggregator
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
r.createInstance
(UserDefinedValueAggregatorDescriptor.java:56)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
r.(UserDefinedValueAggregatorDescriptor.java:75)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
egatorDescriptor(ValueAggregatorJobBase.java:49)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
rDescriptors(ValueAggregatorJobBase.java:60)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
Spec(ValueAggregatorJobBase.java:69)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
lueAggregatorJobBase.java:37)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
Caused by: java.lang.ClassNotFoundException:
com.company.hadoop.metrics.testAggregator
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
r.createInstance
(UserDefinedValueAggregatorDescriptor.java:51)
... 13 more
As near as I can tell I am doing everything correctly. Other code
existing in this jar file is
running just fine - in fact it was used to produce the input to this
test. I've checked my
systems to make I've got the jar file (and XML file) onto each of the 4
compute nodes in the
system.
Can anybody spot my problem or tell me what to do to get this going?
Thanks,
C G
---------------------------------
Tonight's top picks. What will you watch tonight? Preview the hottest
shows on Yahoo! TV.
---------------------------------
Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out.
