Nick,
For jars, you need to use DistributedCache.addFileToClassPath(…). The
archive-adding function is only to be used when you want the file
you're adding to also be extracted into the task's working directory.
Also, you don't need to use DistributedFileSystem classes (its not
supposed to be publicly usable).
Therefore, you may do:
FileSystem fs = FileSystem.get(conf);
DistributedCache.addFileToClassPath(new
Path("hdfs://localhost:9000/my.app/batch.jar"), conf, fs);
Or you may also pack your jar into your driver jar under a lib/
subfolder inside the jar itself (making it an "über" jar, as they call
it), and just submit that.
Either of these approaches will get you going.
On Mon, Apr 9, 2012 at 11:08 PM, Nick Collier <nick.coll...@verizon.net> wrote:
> Hi,
>
> Using Hadoop 1.0.1 I'm trying to use the DistributedCache to add additional
> jars to the classpath used by my Mappers but I can't get it to work. In the
> run(String[] args) method of my Tool implementation, I've tried:
>
> FileSystem fs = DistributedFileSystem.get(conf);
> DistributedCache.addArchiveToClassPath(new
> Path("hdfs://localhost:9000/my.app/batch.jar"), conf, fs);
>
> and
>
> FileSystem fs = DistributedFileSystem.get(conf);
> DistributedCache.addArchiveToClassPath(new Path("/my.app/batch.jar"), conf,
> fs);
>
> and using a local path,
>
> DistributedCache.addArchiveToClassPath(new Path("./lib/batch.jar"), conf);
>
> I've also tried using the addFileToClassPath alternative but also without
> success. I'm running in pseudo-distributed mode. Adding the additional jars
> with -libjars does work, but I would like to get this working as well as it
> will be easier to maintain.
>
> Any help is much appreciated,
>
> Nick
>
>
>
--
Harsh J
