Will-Lo commented on code in PR #3537:
URL: https://github.com/apache/gobblin/pull/3537#discussion_r949567548
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java:
##########
@@ -138,8 +138,13 @@ protected List<FileStatus> getFilesAtPath(FileSystem fs,
Path path, PathFilter f
private List<FileStatus> recursivelyGetFilesAtDatePath(FileSystem fs, Path
path, String traversedDatePath, PathFilter fileFilter,
int level, LocalDateTime startDate, LocalDateTime endDate,
DateTimeFormatter formatter) throws IOException {
List<FileStatus> fileStatuses = Lists.newArrayList();
- Iterator<FileStatus> folderIterator =
Arrays.asList(fs.listStatus(path)).iterator();
-
+ Iterator<FileStatus> folderIterator;
+ try {
+ folderIterator = Arrays.asList(fs.listStatus(path)).iterator();
Review Comment:
By check before list status do you mean to check that all the folders match
the date time format at that current level of granularity? Just following the
previous behavior of this finder, which was to report an exception and return
empty list instead of failing the job. Current issues are caused by failling
the job and not ignoring badly formatted files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
