Will-Lo commented on code in PR #3870:
URL: https://github.com/apache/gobblin/pull/3870#discussion_r1471799258
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -188,11 +215,19 @@ protected Map<Path, FileStatus>
getFilePathsToFileStatus(FileSystem targetFs, Co
this.getFileSetId(), currentSnapshotOverview.getSnapshotId(),
currentSnapshotOverview.getManifestListPath(),
currentSnapshotOverview.getMetadataPath().orElse("<<ERROR:
MISSING!>>"));
- return Maps.newHashMap();
+ TableMetadata readTimeTableMetadata =
currentSnapshotOverview.getTableMetadata().orElseThrow(() -> new
RuntimeException(
+ String.format("~%s~ no table metadata for current snapshot '%s' at
'%s' with metadata path '%s'",
+ this.getFileSetId(), currentSnapshotOverview.getSnapshotId(),
+ currentSnapshotOverview.getManifestListPath(),
+ currentSnapshotOverview.getMetadataPath().orElse("<<ERROR:
MISSING!>>"))));
+ return new GetFilePathsToFileStatusResult(Maps.newHashMap(),
readTimeTableMetadata);
}
+
+ List<TableMetadata> readTimeTableMetadataHolder = Lists.newArrayList(); //
expecting exactly one elem
Iterator<IcebergSnapshotInfo> icebergIncrementalSnapshotInfos =
icebergTable.getIncrementalSnapshotInfosIterator();
Iterator<String> filePathsIterator = Iterators.concat(
Iterators.transform(icebergIncrementalSnapshotInfos, snapshotInfo -> {
+
snapshotInfo.getTableMetadata().ifPresent(readTimeTableMetadataHolder::add);
// log each snapshot, for context, in case of
`FileNotFoundException` during `FileSystem.getFileStatus()`
String manListPath = snapshotInfo.getManifestListPath();
log.info("~{}~ loaded snapshot '{}' at '{}' from metadata path:
'{}'", this.getFileSetId(),
Review Comment:
To understand this PR a bit, we are logging out the paths of each snapshot
here, so we should be able to detect when a replication has mismatched metadata
at this point?
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -126,6 +129,14 @@ protected Iterator<FileSet<CopyEntity>>
createFileSets(FileSystem targetFs, Copy
return Iterators.singletonIterator(fileSet);
}
+ /** Conveys {@link Path}-to-{@link FileStatus} mapping in combination with
{@link TableMetadata} originating from the same atomic read of those paths */
+ @Data
+ @VisibleForTesting
+ protected static final class GetFilePathsToFileStatusResult {
Review Comment:
Does it make sense to call this `IcebergTableFileMetadata`? Wondering if
we're being a bit explicit here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]