gaborkaszab commented on code in PR #17523:
URL: https://github.com/apache/iceberg/pull/17523#discussion_r4547592073


##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO 
io) {
   /**
    * Return the location of this snapshot's manifest list, or null if it is 
not separate.

Review Comment:
   The "or null if.." part is also relevant now here, right?



##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO 
io) {
   /**
    * Return the location of this snapshot's manifest list, or null if it is 
not separate.
    *
-   * @return the location of the manifest list for this Snapshot
+   * <p>Only set for tables with format version 1 through 3. Snapshots in v4 
tables use {@link

Review Comment:
   I'm hesitant adding such a comment here. We also don't mention that for 
`first-row-id`, `added-rows` etc. are only relevant from V3 onwards.
   I'd like it to the spec to describe the relation between this and 
rootManifestLocation.



##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO 
io) {
   /**
    * Return the location of this snapshot's manifest list, or null if it is 
not separate.
    *
-   * @return the location of the manifest list for this Snapshot
+   * <p>Only set for tables with format version 1 through 3. Snapshots in v4 
tables use {@link
+   * #rootManifestLocation()} instead.
+   *
+   * @return the location of the manifest list for this snapshot
    */
   String manifestListLocation();
 
+  /**
+   * Return the location of this snapshot's root manifest, or null if this 
snapshot has no root
+   * manifest.
+   *
+   * <p>v4 snapshots store this in {@code root-manifest} instead of a manifest 
list. At most one of

Review Comment:
   Same comment as above



##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -65,9 +66,14 @@ class BaseSnapshot implements Snapshot {
       Map<String, String> summary,
       Integer schemaId,
       String manifestList,
+      String rootManifest,
       Long firstRowId,
       Long addedRows,
       String keyId) {
+    Preconditions.checkArgument(

Review Comment:
   This verifies they can't be set together. Should we also verify that one of 
them has to be set?



##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -181,7 +189,11 @@ private void cacheManifests(FileIO fileIO) {
     }
 
     if (allManifests == null) {
-      // if manifests isn't set, then the snapshotFile is set and should be 
read to get the list
+      // if manifests isn't set, then the manifest list must be read
+      Preconditions.checkArgument(

Review Comment:
   Don't we want to reverse this and at L181 verify that `rootManifestLocation` 
is null, otherwise throw that we don't support this for V4. We could leave the 
rest of the function intact.



##########
core/src/main/java/org/apache/iceberg/SnapshotParser.java:
##########
@@ -176,9 +180,11 @@ static Snapshot fromJson(JsonNode node) {
 
     String keyId = JsonUtil.getStringOrNull(KEY_ID, node);
 
-    if (node.has(MANIFEST_LIST)) {
-      // the manifest list is stored in a manifest list file
-      String manifestList = JsonUtil.getString(MANIFEST_LIST, node);
+    String manifestList = JsonUtil.getStringOrNull(MANIFEST_LIST, node);
+    String rootManifest = JsonUtil.getStringOrNull(ROOT_MANIFEST, node);
+
+    if (manifestList != null || rootManifest != null) {
+      // manifest-list is used by v1-v3 snapshots, root-manifest by v4 
snapshots

Review Comment:
   nit: comment is not needed



##########
core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java:
##########
@@ -423,6 +425,37 @@ public void testAddSnapshotFromJson() throws IOException {
     assertEquals(action, expected, MetadataUpdateParser.fromJson(json));
   }
 
+  @Test
+  public void testAddSnapshotWithRootManifest() {
+    String action = MetadataUpdateParser.ADD_SNAPSHOT;
+    long snapshotId = 2;
+    int schemaId = 3;

Review Comment:
   nit: this could be inline at L443



##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -241,8 +241,8 @@ private static List<Snapshot> updatePathInSnapshots(
       TableMetadata metadata, String sourcePrefix, String targetPrefix) {
     List<Snapshot> newSnapshots = 
Lists.newArrayListWithCapacity(metadata.snapshots().size());
     for (Snapshot snapshot : metadata.snapshots()) {
-      String newManifestListLocation =
-          newPath(snapshot.manifestListLocation(), sourcePrefix, targetPrefix);
+      String manifestList = snapshot.manifestListLocation();

Review Comment:
   We're introducing the new field to `Snapshot`, I don't think that we should 
implement table rewrite functionality right away. Could that be separate?



##########
core/src/test/java/org/apache/iceberg/TestSnapshotJson.java:
##########
@@ -156,13 +164,16 @@ public void testJsonConversionWithRowLineage() throws 
IOException {
             null,
             null,
             manifestList,
+            null /* rootManifest */,
             firstRowId,
             addedRows,
             null);
     String json = SnapshotParser.toJson(expected);
     Snapshot snapshot = SnapshotParser.fromJson(json);
 
     assertThat(snapshot.snapshotId()).isEqualTo(expected.snapshotId());
+    
assertThat(snapshot.manifestListLocation()).isEqualTo(expected.manifestListLocation());
+    
assertThat(snapshot.rootManifestLocation()).isEqualTo(expected.rootManifestLocation());

Review Comment:
   Not sure what coverage we add with these asserts on rootManifestLocation, 
because they are always null, except in one dedicated test. Should we assert 
`isNull()` as we do with other fields below?



##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -65,9 +66,14 @@ class BaseSnapshot implements Snapshot {
       Map<String, String> summary,
       Integer schemaId,
       String manifestList,
+      String rootManifest,

Review Comment:
   We might want to introduce a dedicated contract once all the other V4 
Snapshot fields are introduced, but let it be the concert of the follow-ups, 
not this one.
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to