[
https://issues.apache.org/jira/browse/GOBBLIN-1714?focusedWorklogId=811662&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-811662
]
ASF GitHub Bot logged work on GOBBLIN-1714:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 23/Sep/22 17:34
Start Date: 23/Sep/22 17:34
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3568:
URL: https://github.com/apache/gobblin/pull/3568#discussion_r978913179
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/RecursiveCopyableDataset.java:
##########
@@ -195,22 +196,28 @@ public Collection<? extends CopyEntity>
getCopyableFiles(FileSystem targetFs, Co
Map<Path, FileStatus> filesInSource =
createPathMap(getFilesAtPath(this.fs, this.rootPath, this.pathFilter),
this.rootPath);
- Map<Path, FileStatus> filesInTarget =
- createPathMap(getFilesAtPath(targetFs, targetPath, this.pathFilter),
targetPath);
+
+ // Allow fileNotFoundException for filesInTarget since if it doesn't
exist, they will be created.
+ List<FileStatus> filesAtPath = Lists.newArrayList();
+ try {
+ filesAtPath = getFilesAtPath(targetFs, targetPath, this.pathFilter);
+ } catch (FileNotFoundException e) {
+ log.info(String.format("Could not find any files on targetFs %s path
%s.", targetFs.getUri(), targetPath));
+ }
+ Map<Path, FileStatus> filesInTarget = createPathMap(filesAtPath,
targetPath);
return getCopyableFilesImpl(configuration, filesInSource, filesInTarget,
targetFs,
nonGlobSearchPath, configuration.getPublishDir(), targetPath);
}
@VisibleForTesting
protected List<FileStatus> getFilesAtPath(FileSystem fs, Path path,
PathFilter fileFilter)
- throws IOException {
+ throws FileNotFoundException {
try {
return FileListUtils
.listFilesToCopyAtPath(fs, path, fileFilter,
applyFilterToDirectories, includeEmptyDirectories);
} catch (IOException e) {
- log.warn(String.format("Could not find any files on fs %s path %s due to
the following exception. Returning an empty list of files.", fs.getUri(),
path), e);
- return Lists.newArrayList();
+ throw new FileNotFoundException(String.format("Could not find any files
on fs %s path %s.", fs.getUri(), path));
}
Review Comment:
Yes, lets handle no workunits creation case separately.
I think we just want to reverse
https://github.com/apache/gobblin/pull/3263/files
Issue Time Tracking
-------------------
Worklog Id: (was: 811662)
Time Spent: 40m (was: 0.5h)
> Silent failure during data copy
> -------------------------------
>
> Key: GOBBLIN-1714
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1714
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Andy Jiang
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
