The .tgz archives are untarred and symlinked to in the working directory. In your code, you mentioned the key to be "package" in the resource description. I'd suggest you try to add files under the "package" dir (actually a symlink). Also, it really depends on how you package really is built.
For more debugging, I'd replace the java command with a shell command and list the files recursively under the current work-dir to stdout. HTH, +Vinod On Tue, Sep 20, 2011 at 10:56 PM, Chris Riccomini <[email protected]>wrote: > Hey All, > > I'm trying to use YARN's LocalResource stuff to ship a tgz file that has > all the jars that I need on my Classpath (for my ApplicationMaster and my > container tasks). I'm having trouble figuring out what my client should put > in the ApplicationMaster exec command that it sends to the Resource manager. > I tried -cp :./*:, but it didn't seem to pick anything up. > > Here are some code snippets: > > // tell resource manager to execute kafka's application master > new ApplicationMasterExecutor(packagePath) > .addCommand("java -cp :./*: kafka.yarn.ApplicationMaster " + > streamerClass + " " + tasks + " " > + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout " > + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr") > .execute(new Configuration) > > When I run that, I see in my log output dir that it could not find > kafka.yarn.ApplicationMaster. I'm the code sample above, packagePath is a > local FS path to my .tgz file. It's uploaded by my ApplicationMasterExecutor > with: > > val packageFile = new File(packagePath); > val packageUrl = > ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualified(new > Path(packagePath))); > > packageResource.setResource(packageUrl); > packageResource.setSize(packageFile.length()); > packageResource.setTimestamp(packageFile.lastModified()); > packageResource.setType(LocalResourceType.ARCHIVE); > packageResource.setVisibility(LocalResourceVisibility.APPLICATION); > > resource.setMemory(memory) > containerCtx.setResource(resource) > containerCtx.setCommands(cmds.toList) > containerCtx.setLocalResources(Collections.singletonMap("package", > packageResource)) > appCtx.setApplicationId(appId) > appCtx.setUser(user.getShortUserName) > appCtx.setAMContainerSpec(containerCtx) > request.setApplicationSubmissionContext(appCtx) > applicationsManager.submitApplication(request) > > When I poke around in /tmp/nm-local-dir/usercache/criccomi, I see nothing > in there after a run. Before, when I was running with visibility set to USER > (instead of APPLICATION), I could see the unzipped data in the filecache, > but that's gone now that I switched to APPLICATION. > > I'm probably just not putting the proper classpath. How do I do this? > > Cheers, > Chris >
