ZihanLi58 commented on code in PR #3598:
URL: https://github.com/apache/gobblin/pull/3598#discussion_r1035389723
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/ManifestBasedDataset.java:
##########
@@ -45,27 +44,31 @@
/**
* A dataset that based on Manifest. We expect the Manifest contains the list
of all the files for this dataset.
* At first phase, we only support copy across different clusters to the same
location. (We can add more feature to support rename in the future)
+ * We will delete the file on target if it's listed in the manifest and not
exist on source
*/
@Slf4j
public class ManifestBasedDataset implements IterableCopyableDataset {
- public static final String CONFIG_PREFIX = CopyConfiguration.COPY_PREFIX +
".manifestBased";
- private static final String DELETE_FILE_NOT_EXIST_ON_SOURCE = CONFIG_PREFIX
+ ".deleteFileNotExistOnSource";
- /** If true, will delete newly empty directories up to the config set in
DELETE_EMPTY_DIRECTORIES_UPTO. */
- public static final String DELETE_EMPTY_DIRECTORIES_KEY = CONFIG_PREFIX +
".deleteEmptyDirectories";
- public static final String DELETE_EMPTY_DIRECTORIES_UPTO = CONFIG_PREFIX +
".deleteEmptyDirectoriesUpTo";
+ private static final String DELETE_FILE_NOT_EXIST_ON_SOURCE =
ManifestBasedDatasetFinder.CONFIG_PREFIX + ".deleteFileNotExistOnSource";
+ /** If true, after delete the file that not exist on source, will delete
newly empty directories up to the config set in DELETE_EMPTY_DIRECTORIES_UPTO.
*/
+ public static final String DELETE_EMPTY_DIRECTORIES_KEY =
ManifestBasedDatasetFinder.CONFIG_PREFIX + ".deleteEmptyDirectories";
+ public static final String DELETE_EMPTY_DIRECTORIES_UPTO =
ManifestBasedDatasetFinder.CONFIG_PREFIX + ".deleteEmptyDirectoriesUpTo";
Review Comment:
I prefer later approach as well as we need to add support for dir sync in
the future. So it makes sense to put the dir that they want to delete in the
manifest, I don't even care about the order as we can order them all together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
