Will-Lo commented on code in PR #3931:
URL: https://github.com/apache/gobblin/pull/3931#discussion_r1573066383
##########
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinClusterConfigurationKeys.java:
##########
@@ -50,6 +50,8 @@ public class GobblinClusterConfigurationKeys {
public static final boolean DEFAULT_STANDALONE_CLUSTER_MODE = false;
// Root working directory for Gobblin cluster
public static final String CLUSTER_WORK_DIR = GOBBLIN_CLUSTER_PREFIX +
"workDir";
+ // Root working dir without appending the application name, keeping
CLUSTER_WORK_DIR property for backward compatibility
+ public static final String CLUSTER_ABSOLUTE_WORK_DIR =
GOBBLIN_CLUSTER_PREFIX + "absolute.workDir";
Review Comment:
It's done by the caller (config file), which can dynamically use properties
in the job to append to the folder. It also can append yarn application ID in
this config file if you so desire by determining the path at runtime. The issue
is that a lot of the other files/folders generated by an E2E Gobblin cluster on
yarn app does not also append the yarn application id (leak of abstraction).
This approach lets there be more explicit control on the caller side to
determine their folder behaviors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
