You should use the DistributedCache.
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#DistributedCache
On Thu, Dec 31, 2009 at 3:21 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Hi,
>
> I have a mapreduce program embeded in a java application, and I am trying
> to
> load additional jar files as add-on for the execution of my map reduce job.
> My program works like this:
>
> JobConf conf = new JobConf(new Configuration(), Demux.class);
> conf.setBoolean("mapred.used.genericoptionsparser",true);
>
> args[x]="-libjars"
> args[x+1]="/path/to/addon.jar";
>
> int res = ToolRunner.run(conf, new Demux(), args);
>
> When the mapreduce job is running, the task failed because it is unable to
> find the additional classes inside addon.jar. If I extract
> /mapredsystem/hadoop/mapredsystem/job_200912171752_25474/job.jar, it looks
> like only the main jar file was in the job.jar. Does the addon.jar needs
> to
> be preloaded into all task tracker node? Could the addon.jar get uploaded
> via the client? If yes, how to do this properly?
>
> I am using Hadoop 0.20.0 rc7 from yahoo with capacity scheduler. Thanks in
> advance.
>
> Regards,
> Eric
>
>
