Will-Lo commented on a change in pull request #3158:
URL: https://github.com/apache/gobblin/pull/3158#discussion_r590865408
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveDataset.java
##########
@@ -130,7 +130,9 @@ public HiveDataset(FileSystem fs, HiveMetastoreClientPool
clientPool, Table tabl
this.tableIdentifier = this.table.getDbName() + "." +
this.table.getTableName();
Path tableLocation = this.table.getPath();
if (!(this.properties.isEmpty())) {
- String datasetStagingDir =
this.properties.getProperty(COPY_TARGET_TABLE_PREFIX_REPLACEMENT) + "/" +
tableLocation.getName();
+ // Will return staging path
+ String datasetStagingDir =
this.properties.getProperty(COPY_TARGET_TABLE_PREFIX_REPLACEMENT) + "/" +
+ tableLocation.getParent().getName() + "/" + tableLocation.getName();
Review comment:
Was thinking about this code more, I think it makes more sense to place
it in `HiveCopyEntityHelper` as it's the class that creates the shards,
registers the path for the hive tables, and calls the path helper to determine
the path from the configurations such as the prefix replacement. Having the
code in the `HiveDataset` class is going to either repeat that logic or miss
edge cases that might've already been handled in with the path helper
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
