autumnust commented on a change in pull request #3050:
URL: https://github.com/apache/incubator-gobblin/pull/3050#discussion_r443061486



##########
File path: 
gobblin-api/src/main/java/org/apache/gobblin/configuration/ConfigurationKeys.java
##########
@@ -207,6 +207,8 @@
   public static final String CLEANUP_OLD_JOBS_DATA = "cleanup.old.job.data";
   public static final boolean DEFAULT_CLEANUP_OLD_JOBS_DATA = false;
   public static final String MAXIMUM_JAR_COPY_RETRY_TIMES_KEY = 
JOB_JAR_FILES_KEY + ".uploading.retry.maximum";
+  public static final String USER_DEFINED_STAGING_DIR = 
"user.defined.staging.dir";

Review comment:
       I'll just name it as "user.defined.static.staging.dir". Let's sync 
offline for why I suggest doing this. 

##########
File path: 
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/writer/FileAwareInputStreamDataWriter.java
##########
@@ -140,9 +140,13 @@ public FileAwareInputStreamDataWriter(State state, int 
numBranches, int branchId
     this.fs = FileSystem.get(uri, conf);
     this.fileContext = FileContext.getFileContext(uri, conf);
 
-    this.stagingDir = this.writerAttemptIdOptional.isPresent() ? WriterUtils
-        .getWriterStagingDir(state, numBranches, branchId, 
this.writerAttemptIdOptional.get())
-        : WriterUtils.getWriterStagingDir(state, numBranches, branchId);
+    if 
(state.getPropAsBoolean(ConfigurationKeys.USER_DEFINED_STAGING_DIR_FLAG)) {

Review comment:
       Think about maintaining the backward-compatibility for such kind of 
change. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Reply via email to