[
https://issues.apache.org/jira/browse/GOBBLIN-712?focusedWorklogId=218938&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-218938
]
ASF GitHub Bot logged work on GOBBLIN-712:
------------------------------------------
Author: ASF GitHub Bot
Created on: 26/Mar/19 21:44
Start Date: 26/Mar/19 21:44
Worklog Time Spent: 10m
Work Description: yukuai518 commented on pull request #2579:
[GOBBLIN-712] Add version strategy pickup for ConfigBasedDataset distcp workflow
URL: https://github.com/apache/incubator-gobblin/pull/2579#discussion_r269328551
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/replication/ConfigBasedDataset.java
##########
@@ -177,12 +231,30 @@ public String datasetURN() {
watermarkMetadataCopied = true;
}
- // skip copy same file
- if (copyToFileMap.containsKey(newPath) &&
copyToFileMap.get(newPath).getLen() == originFileStatus.getLen()
- && copyToFileMap.get(newPath).getModificationTime() >
originFileStatus.getModificationTime()) {
- log.debug("Copy from timestamp older than copy to timestamp, skipped
copy {} for dataset with metadata {}",
- originFileStatus.getPath(), this.rc.getMetaData());
- } else {
+
+ boolean shouldCopy = true;
+ if (copyToFileMap.containsKey(newPath)) {
+ if (this.srcDataFileVersionStrategy.get() instanceof
ModTimeDataFileVersionStrategy) {
+ // default version strategy should also compare data file length
+ if (copyToFileMap.get(newPath).getLen() == originFileStatus.getLen()
+ && copyToFileMap.get(newPath).getModificationTime() >
originFileStatus.getModificationTime()) {
+ log.debug("Copy from timestamp older than copy to timestamp,
skipped copy {} for dataset with metadata {}",
+ originFileStatus.getPath(), this.rc.getMetaData());
+ shouldCopy = false;
+ }
+ } else {
+ // other version strategy should just compare the version number
+ Comparable srcVer =
this.srcDataFileVersionStrategy.get().getVersion(originFileStatus.getPath());
+ Comparable dstVer =
this.dstDataFileVersionStrategy.get().getVersion(copyToFileMap.get(newPath).getPath());
+ if (srcVer.compareTo(dstVer) == 0) {
Review comment:
I think so, let me change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 218938)
Time Spent: 1h 10m (was: 1h)
> Add version strategy for configbased dataset copy
> -------------------------------------------------
>
> Key: GOBBLIN-712
> URL: https://issues.apache.org/jira/browse/GOBBLIN-712
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Kuai Yu
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
