Will-Lo commented on a change in pull request #3158:
URL: https://github.com/apache/gobblin/pull/3158#discussion_r594904025
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/publisher/CopyDataPublisher.java
##########
@@ -117,8 +117,9 @@ public CopyDataPublisher(State state) throws IOException {
this.fs = FileSystem.get(URI.create(uri),
WriterUtils.getFsConfiguration(state));
FileAwareInputStreamDataWriterBuilder.setJobSpecificOutputPaths(state);
-
- this.writerOutputDir = new
Path(state.getProp(ConfigurationKeys.WRITER_OUTPUT_DIR));
+ // If directories are sharded by dataset, initialize writers using
workunit state instead
+ this.writerOutputDir =
state.getPropAsBoolean(ConfigurationKeys.USE_DATASET_LOCAL_WORK_DIR) ?
Review comment:
It's only in the shard-specific handler where this is being set, this
check is for the pre-existing case where the writer/publisher would propagate
the job configuration for path setting to the writer/publisher, which the shard
logic wants to avoid. We could create a default handler that performs this
logic. I think it may be the way forward considering that this logic is
repeated in the writer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
