On 22/03/11 13:34, Andy Doddington wrote:
I am trying to create a bundled jar file for running using the hadoop ‘jar’
command. However, when I try to do this it fails to
find the jar files and other resources that I have placed into the jar (pointed
at by the Class-Path property in the MANIFEST.MF).
I have unpacked the jar file into a temporary directory and run it manually
using the java -jar command (after manually listing
the various hadoop jar files that are required in the classpath) and it works
fine in this mode.
I have read on some places that the workaround is to put my jar files and other
resources into the hadoop ‘lib’ directory, but
I really don’t want to do this, since it feels horribly kludgy.
Putting them in lib/ is bad as you need to restart the cluster to get
changes out, and you can't have jobs with different versions in them
use the -libjars option instead
I don’t mind creating a ‘lib’ directory in my jar file, since this will be
hidden, but I would appreciate more documentation as to why/how I need to do
this.
because the class-path manifest thing is designed for classloaders in
the local filesystem where relative paths are used to find JARs and
everything runs locally, and is only handled by the normal Java main
entry point and the applet loader.
Can anybody advise as to how I can get 'hadoop jar’ to honour the Class-Path
entry in the MANIFEST.MF?
You'd probably have to write the code to
-parse the entry at job submission
-find the JARs or fail
-somehow include all these JARs in the list of dependencies for the
job submission list via a transform to the -libjars command.
-add the tests for this
I'm not sure it's worth the effort.
