Hi My use include custom AM and launching own containers. Like I mentioned before LocalResources does provide me a facility to add jars and files (No problem in that). I was just curious, is there any other way I am missing or am on a right track.
On Tue, Jul 16, 2013 at 3:52 AM, Omkar Joshi <[email protected]> wrote: > Hi, > > Looking at your use case you need you jar files to be accessible on node > manager. > > If you are using Map reduce on YARN then the way you can achieve this is. > 1) job.addCacheFile() ... to add simple files. > 2) job.addCacheFile()... for ".jar", ".tar", ".zip" files. Note. Node > manager will extract them for you once it has downloaded them on node > manager. > Now you can access these files from your current working directory in node > manager where your container will run. So if you want to access them ..then > you can access like > > File f = new File("./<file-name>"); > One thing to be remembered here is that only symlink will be present in the > current working directory and it will point to the localized (downloaded) > file in application cache. (try f.getAbsolutePath()). > > In case you are launching your own containers ... through your own > application master then you should find these symlinks in your current > working directory once container starts. All the resources (ex. jar files) > which you want node manager to download (localize) prior to start can be > specified in the ContainerLaunchContext at startContainer call. > > Thanks, > Omkar Joshi > *Hortonworks Inc.* <http://www.hortonworks.com> > > > On Fri, Jul 12, 2013 at 5:51 PM, Azuryy Yu <[email protected]> wrote: > > > I think that's true. > > On Jul 13, 2013 2:03 AM, "Kapoor" <[email protected]> wrote: > > > > > Thanks for the link mate > > > So I have to use LocalResource to achieve the distributed cache > behaviour > > > on YARN? > > > On 12-Jul-2013 11:25 PM, "Omkar Joshi" <[email protected]> wrote: > > > > > > > https://issues.apache.org/jira/browse/MAPREDUCE-5385 > > > > > > > > > > > > > > > > Thanks, > > > > Omkar Joshi > > > > *Hortonworks Inc.* <http://www.hortonworks.com> > > > > > > > > > > > > On Fri, Jul 12, 2013 at 4:44 AM, Azuryy Yu <[email protected]> > wrote: > > > > > > > > > DistributedCache is not deprecated in the hadoop-2.x, map/reduce > job > > > can > > > > > also use it as before. but for YARN task(not MR), only > LocalResource > > is > > > > > recognized, no DistributedCache. > > > > > > > > > > please correct me if I am wrong. > > > > > > > > > > > > > > > On Fri, Jul 12, 2013 at 1:54 PM, Kapoor <[email protected]> > wrote: > > > > > > > > > > > It looks like DistributedCache is deprecated in latest code. > > > > > > Do we have any alternative of DistributedCache in apies ? > > > > > > I know I can use LocalResource in YARN. > > > > > > > > > > > > My use case is, I need some files (jar) from local machine to be > > > > > available > > > > > > to datanodes, so that running container can have access to it in > > > their > > > > > > classpath. > > > > > > > > > > > > > > > > > > > > >
