As Vinod mentioned, package your dirs so that they untar into the CWD.
If you have a directory structure in your .tgz (e.g. lib.tgz -> lib/foo.jar)
you'll need to set your classpath to be ./foo/*.
hth,
Arun
On Sep 20, 2011, at 10:26 AM, Chris Riccomini wrote:
> Hey All,
>
> I'm trying to use YARN's LocalResource stuff to ship a tgz file that has all
> the jars that I need on my Classpath (for my ApplicationMaster and my
> container tasks). I'm having trouble figuring out what my client should put
> in the ApplicationMaster exec command that it sends to the Resource manager.
> I tried -cp :./*:, but it didn't seem to pick anything up.
>
> Here are some code snippets:
>
> // tell resource manager to execute kafka's application master
> new ApplicationMasterExecutor(packagePath)
> .addCommand("java -cp :./*: kafka.yarn.ApplicationMaster " +
> streamerClass + " " + tasks + " "
> + "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout "
> + "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
> .execute(new Configuration)
>
> When I run that, I see in my log output dir that it could not find
> kafka.yarn.ApplicationMaster. I'm the code sample above, packagePath is a
> local FS path to my .tgz file. It's uploaded by my ApplicationMasterExecutor
> with:
>
> val packageFile = new File(packagePath);
> val packageUrl =
> ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualified(new
> Path(packagePath)));
>
> packageResource.setResource(packageUrl);
> packageResource.setSize(packageFile.length());
> packageResource.setTimestamp(packageFile.lastModified());
> packageResource.setType(LocalResourceType.ARCHIVE);
> packageResource.setVisibility(LocalResourceVisibility.APPLICATION);
>
> resource.setMemory(memory)
> containerCtx.setResource(resource)
> containerCtx.setCommands(cmds.toList)
> containerCtx.setLocalResources(Collections.singletonMap("package",
> packageResource))
> appCtx.setApplicationId(appId)
> appCtx.setUser(user.getShortUserName)
> appCtx.setAMContainerSpec(containerCtx)
> request.setApplicationSubmissionContext(appCtx)
> applicationsManager.submitApplication(request)
>
> When I poke around in /tmp/nm-local-dir/usercache/criccomi, I see nothing in
> there after a run. Before, when I was running with visibility set to USER
> (instead of APPLICATION), I could see the unzipped data in the filecache, but
> that's gone now that I switched to APPLICATION.
>
> I'm probably just not putting the proper classpath. How do I do this?
>
> Cheers,
> Chris
