Will-Lo commented on code in PR #3870:
URL: https://github.com/apache/gobblin/pull/3870#discussion_r1471799258


##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -188,11 +215,19 @@ protected Map<Path, FileStatus> 
getFilePathsToFileStatus(FileSystem targetFs, Co
           this.getFileSetId(), currentSnapshotOverview.getSnapshotId(),
           currentSnapshotOverview.getManifestListPath(),
           currentSnapshotOverview.getMetadataPath().orElse("<<ERROR: 
MISSING!>>"));
-      return Maps.newHashMap();
+      TableMetadata readTimeTableMetadata = 
currentSnapshotOverview.getTableMetadata().orElseThrow(() -> new 
RuntimeException(
+          String.format("~%s~ no table metadata for current snapshot '%s' at 
'%s' with metadata path '%s'",
+              this.getFileSetId(), currentSnapshotOverview.getSnapshotId(),
+              currentSnapshotOverview.getManifestListPath(),
+              currentSnapshotOverview.getMetadataPath().orElse("<<ERROR: 
MISSING!>>"))));
+      return new GetFilePathsToFileStatusResult(Maps.newHashMap(), 
readTimeTableMetadata);
     }
+
+    List<TableMetadata> readTimeTableMetadataHolder = Lists.newArrayList(); // 
expecting exactly one elem
     Iterator<IcebergSnapshotInfo> icebergIncrementalSnapshotInfos = 
icebergTable.getIncrementalSnapshotInfosIterator();
     Iterator<String> filePathsIterator = Iterators.concat(
         Iterators.transform(icebergIncrementalSnapshotInfos, snapshotInfo -> {
+          
snapshotInfo.getTableMetadata().ifPresent(readTimeTableMetadataHolder::add);
           // log each snapshot, for context, in case of 
`FileNotFoundException` during `FileSystem.getFileStatus()`
           String manListPath = snapshotInfo.getManifestListPath();
           log.info("~{}~ loaded snapshot '{}' at '{}' from metadata path: 
'{}'", this.getFileSetId(),

Review Comment:
   To understand this PR a bit, we are logging out the paths of each snapshot 
here, so we should be able to detect when a replication has mismatched metadata 
at this point?



##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDataset.java:
##########
@@ -126,6 +129,14 @@ protected Iterator<FileSet<CopyEntity>> 
createFileSets(FileSystem targetFs, Copy
     return Iterators.singletonIterator(fileSet);
   }
 
+  /** Conveys {@link Path}-to-{@link FileStatus} mapping in combination with 
{@link TableMetadata} originating from the same atomic read of those paths */
+  @Data
+  @VisibleForTesting
+  protected static final class GetFilePathsToFileStatusResult {

Review Comment:
   Does it make sense to call this `IcebergTableFileMetadata`? Wondering if 
we're being a bit explicit here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to