Using LocalResource is the right thing to use to pass any resources your application needs to also make available locally. DistributedCache is "simpler" but is restricted for reuse under MR APIs right now. Some further abstraction ideas to combine best of both lands is definitely welcome.
On Tue, Jul 16, 2013 at 12:00 PM, Kapoor <[email protected]> wrote: > Hi > > My use include custom AM and launching own containers. > Like I mentioned before LocalResources does provide me a facility to add > jars and files (No problem in that). > I was just curious, is there any other way I am missing or am on a right > track. > > > On Tue, Jul 16, 2013 at 3:52 AM, Omkar Joshi <[email protected]> wrote: > >> Hi, >> >> Looking at your use case you need you jar files to be accessible on node >> manager. >> >> If you are using Map reduce on YARN then the way you can achieve this is. >> 1) job.addCacheFile() ... to add simple files. >> 2) job.addCacheFile()... for ".jar", ".tar", ".zip" files. Note. Node >> manager will extract them for you once it has downloaded them on node >> manager. >> Now you can access these files from your current working directory in node >> manager where your container will run. So if you want to access them ..then >> you can access like >> >> File f = new File("./<file-name>"); >> One thing to be remembered here is that only symlink will be present in the >> current working directory and it will point to the localized (downloaded) >> file in application cache. (try f.getAbsolutePath()). >> >> In case you are launching your own containers ... through your own >> application master then you should find these symlinks in your current >> working directory once container starts. All the resources (ex. jar files) >> which you want node manager to download (localize) prior to start can be >> specified in the ContainerLaunchContext at startContainer call. >> >> Thanks, >> Omkar Joshi >> *Hortonworks Inc.* <http://www.hortonworks.com> >> >> >> On Fri, Jul 12, 2013 at 5:51 PM, Azuryy Yu <[email protected]> wrote: >> >> > I think that's true. >> > On Jul 13, 2013 2:03 AM, "Kapoor" <[email protected]> wrote: >> > >> > > Thanks for the link mate >> > > So I have to use LocalResource to achieve the distributed cache >> behaviour >> > > on YARN? >> > > On 12-Jul-2013 11:25 PM, "Omkar Joshi" <[email protected]> wrote: >> > > >> > > > https://issues.apache.org/jira/browse/MAPREDUCE-5385 >> > > > >> > > > >> > > > >> > > > Thanks, >> > > > Omkar Joshi >> > > > *Hortonworks Inc.* <http://www.hortonworks.com> >> > > > >> > > > >> > > > On Fri, Jul 12, 2013 at 4:44 AM, Azuryy Yu <[email protected]> >> wrote: >> > > > >> > > > > DistributedCache is not deprecated in the hadoop-2.x, map/reduce >> job >> > > can >> > > > > also use it as before. but for YARN task(not MR), only >> LocalResource >> > is >> > > > > recognized, no DistributedCache. >> > > > > >> > > > > please correct me if I am wrong. >> > > > > >> > > > > >> > > > > On Fri, Jul 12, 2013 at 1:54 PM, Kapoor <[email protected]> >> wrote: >> > > > > >> > > > > > It looks like DistributedCache is deprecated in latest code. >> > > > > > Do we have any alternative of DistributedCache in apies ? >> > > > > > I know I can use LocalResource in YARN. >> > > > > > >> > > > > > My use case is, I need some files (jar) from local machine to be >> > > > > available >> > > > > > to datanodes, so that running container can have access to it in >> > > their >> > > > > > classpath. >> > > > > > >> > > > > >> > > > >> > > >> > >> -- Harsh J
