ZihanLi58 commented on code in PR #3537:
URL: https://github.com/apache/gobblin/pull/3537#discussion_r949563641
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java:
##########
@@ -138,8 +138,13 @@ protected List<FileStatus> getFilesAtPath(FileSystem fs,
Path path, PathFilter f
private List<FileStatus> recursivelyGetFilesAtDatePath(FileSystem fs, Path
path, String traversedDatePath, PathFilter fileFilter,
int level, LocalDateTime startDate, LocalDateTime endDate,
DateTimeFormatter formatter) throws IOException {
List<FileStatus> fileStatuses = Lists.newArrayList();
- Iterator<FileStatus> folderIterator =
Arrays.asList(fs.listStatus(path)).iterator();
-
+ Iterator<FileStatus> folderIterator;
+ try {
+ folderIterator = Arrays.asList(fs.listStatus(path)).iterator();
Review Comment:
why not add check before list status but rely on the exception?
what happens if it's some error on the fs side? isn't it better for us to
throw exception to fail the job instead of returning empty list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
