Hi,
Looking at your use case you need you jar files to be accessible on node
manager.
If you are using Map reduce on YARN then the way you can achieve this is.
1) job.addCacheFile() ... to add simple files.
2) job.addCacheFile()... for ".jar", ".tar", ".zip" files. Note. Node
manager will extract them for you once it has downloaded them on node
manager.
Now you can access these files from your current working directory in node
manager where your container will run. So if you want to access them ..then
you can access like
File f = new File("./<file-name>");
One thing to be remembered here is that only symlink will be present in the
current working directory and it will point to the localized (downloaded)
file in application cache. (try f.getAbsolutePath()).
In case you are launching your own containers ... through your own
application master then you should find these symlinks in your current
working directory once container starts. All the resources (ex. jar files)
which you want node manager to download (localize) prior to start can be
specified in the ContainerLaunchContext at startContainer call.
Thanks,
Omkar Joshi
*Hortonworks Inc.* <http://www.hortonworks.com>
On Fri, Jul 12, 2013 at 5:51 PM, Azuryy Yu <[email protected]> wrote:
> I think that's true.
> On Jul 13, 2013 2:03 AM, "Kapoor" <[email protected]> wrote:
>
> > Thanks for the link mate
> > So I have to use LocalResource to achieve the distributed cache behaviour
> > on YARN?
> > On 12-Jul-2013 11:25 PM, "Omkar Joshi" <[email protected]> wrote:
> >
> > > https://issues.apache.org/jira/browse/MAPREDUCE-5385
> > >
> > >
> > >
> > > Thanks,
> > > Omkar Joshi
> > > *Hortonworks Inc.* <http://www.hortonworks.com>
> > >
> > >
> > > On Fri, Jul 12, 2013 at 4:44 AM, Azuryy Yu <[email protected]> wrote:
> > >
> > > > DistributedCache is not deprecated in the hadoop-2.x, map/reduce job
> > can
> > > > also use it as before. but for YARN task(not MR), only LocalResource
> is
> > > > recognized, no DistributedCache.
> > > >
> > > > please correct me if I am wrong.
> > > >
> > > >
> > > > On Fri, Jul 12, 2013 at 1:54 PM, Kapoor <[email protected]> wrote:
> > > >
> > > > > It looks like DistributedCache is deprecated in latest code.
> > > > > Do we have any alternative of DistributedCache in apies ?
> > > > > I know I can use LocalResource in YARN.
> > > > >
> > > > > My use case is, I need some files (jar) from local machine to be
> > > > available
> > > > > to datanodes, so that running container can have access to it in
> > their
> > > > > classpath.
> > > > >
> > > >
> > >
> >
>
