[
https://issues.apache.org/jira/browse/GOBBLIN-1809?focusedWorklogId=856272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-856272
]
ASF GitHub Bot logged work on GOBBLIN-1809:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Apr/23 22:11
Start Date: 11/Apr/23 22:11
Worklog Time Spent: 10m
Work Description: jack-moseley commented on code in PR #3670:
URL: https://github.com/apache/gobblin/pull/3670#discussion_r1163382958
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/version/finder/LookbackDateTimeDatasetVersionFinder.java:
##########
@@ -67,25 +66,20 @@ public LookbackDateTimeDatasetVersionFinder(FileSystem fs,
Config config, Instan
Preconditions.checkArgument(config.hasPath(VERSION_LOOKBACK_PERIOD) ,
"Missing required property " + VERSION_LOOKBACK_PERIOD);
PeriodFormatter periodFormatter =
new
PeriodFormatterBuilder().appendYears().appendSuffix("y").appendMonths().appendSuffix("M").appendDays()
-
.appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
- this.stepDuration = Duration.standardMinutes(1);
+ .appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+ this.stepDuration = Duration.standardHours(1);
this.pathPrefix = ConfigUtils.getString(config, VERSION_PATH_PREFIX, "");
this.lookbackPeriod =
periodFormatter.parsePeriod(config.getString(VERSION_LOOKBACK_PERIOD));
this.endTime = endTime;
}
- @Override
- public Class<? extends FileSystemDatasetVersion> versionClass() {
- return TimestampedDatasetVersion.class;
- }
-
@Override
public Collection<TimestampedDatasetVersion> findDatasetVersions(Dataset
dataset) throws IOException {
FileSystemDataset fsDataset = (FileSystemDataset) dataset;
Set<TimestampedDatasetVersion> versions = new HashSet<>();
Instant startTime = endTime.minus(lookbackPeriod.toStandardDuration());
- while (startTime.isBefore(endTime)) {
+ while (!startTime.isAfter(endTime)) {
Review Comment:
Yeah I think it does look better, updated it.
Issue Time Tracking
-------------------
Worklog Id: (was: 856272)
Time Spent: 2h 20m (was: 2h 10m)
> Add new lookback version finder for use with iceberg retention
> --------------------------------------------------------------
>
> Key: GOBBLIN-1809
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1809
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
