[GitHub] [gobblin] arjun4084346 commented on a diff in pull request #3563: [GOBBLIN-1708] Improve TimeAwareRecursiveCopyableDataset to lookback only into datefolders that match range

[GitBox]

arjun4084346 commented on code in PR #3563:
URL: https://github.com/apache/gobblin/pull/3563#discussion_r974575775


##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java:
##########
@@ -134,9 +134,41 @@ protected List<FileStatus> getFilesAtPath(FileSystem fs, 
Path path, PathFilter f
     return recursivelyGetFilesAtDatePath(fs, path, "", fileFilter, 1, 
startDate, endDate, formatter);
   }
 
+  /**
+   * Checks if the datePath provided is in the range of the start and end 
dates.
+   * Rounds startDate and endDate to the same granularity as datePath prior to 
comparing.
+   * Returns true if the datePath provided is in the range of start and end 
dates, inclusive.
+   * @param startDate
+   * @param endDate
+   * @param datePath
+   * @return true/false
+   */
+  public Boolean checkPathDateTimeValidity(LocalDateTime startDate, 
LocalDateTime endDate, String datePath) {
+    boolean onlyNumbers = datePath.matches("^[0-9/]+$");
+    if (onlyNumbers) {
+      String allGranularity = "yyyy/MM/dd/HH/mm/ss/SSS";
+      String roundGranularity = allGranularity.substring(0, datePath.length());
+      DateTimeFormatter formatGranularity = 
DateTimeFormat.forPattern(roundGranularity);
+
+      LocalDateTime traversedDatePathRound = 
formatGranularity.parseLocalDateTime(datePath);
+      LocalDateTime startDateRound = 
formatGranularity.parseLocalDateTime(startDate.toString(roundGranularity));
+      LocalDateTime endDateRound = 
formatGranularity.parseLocalDateTime(endDate.toString(roundGranularity));
+
+      boolean afterOrOnStartDate = 
traversedDatePathRound.isAfter(startDateRound) || 
traversedDatePathRound.isEqual(startDateRound);
+      boolean beforeOrOnEndDate = 
traversedDatePathRound.isBefore(endDateRound) || 
traversedDatePathRound.isEqual(endDateRound);
+      return afterOrOnStartDate && beforeOrOnEndDate;
+    }
+    return false;

Review Comment:
   Should it return true?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org