Will-Lo commented on code in PR #4030:
URL: https://github.com/apache/gobblin/pull/4030#discussion_r1755520391
##########
gobblin-temporal/src/main/java/org/apache/gobblin/temporal/yarn/YarnService.java:
##########
@@ -484,12 +487,29 @@ private void requestContainer(Optional<String>
preferredNode, Resource resource)
protected ContainerLaunchContext newContainerLaunchContext(ContainerInfo
containerInfo)
throws IOException {
Path appWorkDir =
GobblinClusterUtils.getAppWorkDirPathFromConfig(this.config, this.fs,
this.applicationName, this.applicationId);
+ Path containerJarsUnsharedDir = new Path(appWorkDir,
GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+ Path jarCacheDir = this.jarCacheEnabled ?
YarnHelixUtils.getJarPathCacheAndCleanIfNeeded(this.config, this.fs) :
appWorkDir;
+ Path containerJarsCachedDir = new Path(jarCacheDir,
GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
+ LOGGER.info("Container cached jars root dir: " + containerJarsCachedDir);
+ LOGGER.info("Container uncached jars root dir: " +
containerJarsUnsharedDir);
Path containerWorkDir = new Path(appWorkDir,
GobblinYarnConfigurationKeys.CONTAINER_WORK_DIR_NAME);
- Map<String, LocalResource> resourceMap = Maps.newHashMap();
+ Map<String, LocalResource> resourceMap = Maps.newHashMap();
+ // Always fetch any jars from the appWorkDir for any potential snapshot
jars
addContainerLocalResources(new Path(appWorkDir,
GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
- addContainerLocalResources(new Path(containerWorkDir,
GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME), resourceMap);
+ if
(this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {
+ addContainerLocalResources(new Path(containerJarsUnsharedDir,
GobblinYarnConfigurationKeys.APP_JARS_DIR_NAME),
+ resourceMap);
+ }
+ if (this.jarCacheEnabled) {
+ addContainerLocalResources(new Path(jarCacheDir,
GobblinYarnConfigurationKeys.LIB_JARS_DIR_NAME), resourceMap);
+ if
(this.config.hasPath(GobblinYarnConfigurationKeys.CONTAINER_FILES_LOCAL_KEY)) {
Review Comment:
Oops should have used `gobblin.yarn.container.jars` key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
