phet commented on code in PR #3931:
URL: https://github.com/apache/gobblin/pull/3931#discussion_r1573014909
##########
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java:
##########
@@ -50,6 +50,8 @@ public class GobblinClusterConfigurationKeys {
public static final boolean DEFAULT_STANDALONE_CLUSTER_MODE = false;
// Root working directory for Gobblin cluster
public static final String CLUSTER_WORK_DIR = GOBBLIN_CLUSTER_PREFIX +
"workDir";
+ // Root working dir without appending the application name, keeping
CLUSTER_WORK_DIR property for backward compatibility
+ public static final String CLUSTER_ABSOLUTE_WORK_DIR =
GOBBLIN_CLUSTER_PREFIX + "absolute.workDir";
Review Comment:
two thoughts here:
1. `gobblin.cluster.workDir` is already typically an absolute path. maybe
this would be `gobblin.cluster.exact.workDir`
2. not appending the job/app name seems problematic with the potential for
cross-job collisions (e.g. if two different jobs by the same user intend to use
the same exact dir). isn't there a danger there?
##########
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterUtils.java:
##########
@@ -254,4 +257,13 @@ public static FileSystem buildFileSystem(Config config,
Configuration conf)
.get(URI.create(config.getString(ConfigurationKeys.FS_URI_KEY)), conf)
: FileSystem.get(conf);
}
+
+ public static FileSystem buildNewInstanceFileSystem(Config config,
Configuration conf) throws IOException {
Review Comment:
`ddm.work.assistance.Help` already has a number of varieties of
`loadFileSystem`. this may not duplicate any of the ones there right now, but
should we anyway define it over there and look into the possibility of merging
this w/ one of those other impls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
