Hey All,
I'm trying to use YARN's LocalResource stuff to ship a tgz file that has all
the jars that I need on my Classpath (for my ApplicationMaster and my container
tasks). I'm having trouble figuring out what my client should put in the
ApplicationMaster exec command that it sends to the Resource manager. I tried
-cp :./*:, but it didn't seem to pick anything up.
Here are some code snippets:
// tell resource manager to execute kafka's application master
new ApplicationMasterExecutor(packagePath)
.addCommand("java -cp :./*: kafka.yarn.ApplicationMaster " +
streamerClass + " " + tasks + " "
+ "1>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stdout "
+ "2>" + ApplicationConstants.LOG_DIR_EXPANSION_VAR + "/stderr")
.execute(new Configuration)
When I run that, I see in my log output dir that it could not find
kafka.yarn.ApplicationMaster. I'm the code sample above, packagePath is a local
FS path to my .tgz file. It's uploaded by my ApplicationMasterExecutor with:
val packageFile = new File(packagePath);
val packageUrl =
ConverterUtils.getYarnUrlFromPath(FileContext.getFileContext.makeQualified(new
Path(packagePath)));
packageResource.setResource(packageUrl);
packageResource.setSize(packageFile.length());
packageResource.setTimestamp(packageFile.lastModified());
packageResource.setType(LocalResourceType.ARCHIVE);
packageResource.setVisibility(LocalResourceVisibility.APPLICATION);
resource.setMemory(memory)
containerCtx.setResource(resource)
containerCtx.setCommands(cmds.toList)
containerCtx.setLocalResources(Collections.singletonMap("package",
packageResource))
appCtx.setApplicationId(appId)
appCtx.setUser(user.getShortUserName)
appCtx.setAMContainerSpec(containerCtx)
request.setApplicationSubmissionContext(appCtx)
applicationsManager.submitApplication(request)
When I poke around in /tmp/nm-local-dir/usercache/criccomi, I see nothing in
there after a run. Before, when I was running with visibility set to USER
(instead of APPLICATION), I could see the unzipped data in the filecache, but
that's gone now that I switched to APPLICATION.
I'm probably just not putting the proper classpath. How do I do this?
Cheers,
Chris
