If you are using Maven to build your code, I'd recommend using their assembly
plugin. Just add this to your <plugins> part of your projects pom, and run "mvn
package".
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance
merges -->
<phase>package</phase> <!-- append to the packaging
phase. -->
<goals>
<goal>single</goal> <!-- goals == mojos -->
</goals>
</execution>
</executions>
</plugin>
You'll end up with a huge jar called foo-with-dependencies.jar (or similar)
that you can use to run your job.
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
[email protected] - 702-943-7833
On Feb 21, 2011, at 7:11 PM, Zhengguo 'Mike' SUN wrote:
> OK. I gave up on my code and switched to run some example code from Mahout. I
> ran the clustering example from quickstart as follow:
>
> bin/hadoop jar ../trunk/examples/target/mahout-examples-0.4-SNAPSHOT.job
> org.apache.mahout.clustering.syntheticcontrol.canopy.Job
>
> Again, it worked on my local machine and gave me ClassNotFoundException in
> the cluster.
>
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.syntheticcontrol.canopy.InputMapper
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:919)
> at org.apache.hadoop.mapreduce.JobContext.getMapperClass(JobContext.java:161)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:532)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.mahout.clustering.syntheticcontrol.canopy.InputMapper
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:866)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:917)
> ... 4 more
>
>
>
>
> From: Sean Owen <[email protected]>
> To: Zhengguo 'Mike' SUN <[email protected]>
> Cc: "[email protected]" <[email protected]>
> Sent: Monday, February 21, 2011 3:34 PM
> Subject: Re: LanczosSolver and ClassNotFoundException
>
> In contrast, what is the scenario where it didn't work then?
> This may be a dumb question that a Hadoop guru would know better.
> MapTask.run() occurs in the... master? but not on the "client side". I
> don't know why the classpath would be different though, but, this is
> the essential issue.
>
> (I've not seen this ever.)
>
> On Mon, Feb 21, 2011 at 7:52 PM, Zhengguo 'Mike' SUN
> <[email protected]> wrote:
>> Yes. I put the .jar files under the lib/ directory of my own jar. I didn't
>> set the Class-Path in MANIFEST.MF in the past. I just tried putting all jar
>> files in the Class-Path of MANIFEST.MF, it still gave me the same exception.
>> My experiment is a little bit different. It creates a Vector in the map
>> function of the Mapper class. It worked correctly in the cluster.
>>
>
>
