stevenzwu commented on code in PR #18924:
URL: https://github.com/apache/iceberg/pull/18924#discussion_r4568274100


##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -180,6 +180,15 @@ default String rootManifestLocation() {
     return null;
   }
 
+  /**
+   * Return the size in bytes of this snapshot's root manifest, or null if it 
has no root manifest.
+   *
+   * @return the size in bytes of the root manifest for this snapshot
+   */
+  default Long rootSizeInBytes() {

Review Comment:
   nit: `rootManifestSizeInBytes()`? It pairs well with 
`rootManifestLocation()`, and "root" alone is a bit ambiguous on `Snapshot`. 
The JSON key can become `root-manifest-size-in-bytes` for consistency.
   



##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -78,6 +80,16 @@ class BaseSnapshot implements Snapshot {
         manifestList != null || rootManifest != null,
         "Snapshot %s must have a manifest list or a root manifest",
         snapshotId);
+    Preconditions.checkArgument(
+        rootManifest == null || rootSizeInBytes != null,
+        "Root manifest size is required when root manifest is set");
+    Preconditions.checkArgument(
+        rootManifest != null || rootSizeInBytes == null,
+        "Root manifest size is not allowed if root manifest is null");
+    Preconditions.checkArgument(
+        rootSizeInBytes == null || rootSizeInBytes >= 0,
+        "Invalid root manifest size (cannot be negative): %s",
+        rootSizeInBytes);

Review Comment:
   nit: the required and non-negative checks could be merged:
   
   ```suggestion
       Preconditions.checkArgument(
           rootManifest == null || (rootSizeInBytes != null && rootSizeInBytes 
>= 0),
           "Invalid root manifest size for snapshot %s: %s",
           snapshotId,
           rootSizeInBytes);
       Preconditions.checkArgument(
           rootManifest != null || rootSizeInBytes == null,
           "Snapshot %s cannot have a root manifest size without a root 
manifest",
           snapshotId);
   ```
   



##########
core/src/test/java/org/apache/iceberg/TestSnapshot.java:
##########
@@ -74,14 +74,82 @@ void snapshotWithRootManifestExposesLocation() {
             1 /* schemaId */,
             null /* manifestList */,
             "root-manifest.parquet" /* rootManifest */,
+            1024L /* rootSizeInBytes */,
             10L /* firstRowId */,
             5L /* addedRows */,
             "key-1" /* keyId */);
 
     
assertThat(snapshot.rootManifestLocation()).isEqualTo("root-manifest.parquet");
+    assertThat(snapshot.rootSizeInBytes()).isEqualTo(1024L);
     assertThat(snapshot.manifestListLocation()).isNull();
   }
 
+  @TestTemplate
+  void rejectsRootManifestWithoutSize() {
+    assertThatThrownBy(
+            () ->
+                new BaseSnapshot(
+                    0 /* sequenceNumber */,
+                    1 /* snapshotId */,
+                    null /* parentId */,
+                    0 /* timestampMillis */,
+                    DataOperations.APPEND,
+                    null /* summary */,
+                    1 /* schemaId */,
+                    null /* manifestList */,
+                    "root-manifest.parquet" /* rootManifest */,
+                    null /* rootSizeInBytes */,
+                    10L /* firstRowId */,
+                    5L /* addedRows */,
+                    "key-1" /* keyId */))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Root manifest size is required when root manifest is 
set");
+  }
+
+  @TestTemplate
+  void rejectsRootSizeInBytesWithoutRootManifest() {
+    assertThatThrownBy(
+            () ->
+                new BaseSnapshot(
+                    0 /* sequenceNumber */,
+                    1 /* snapshotId */,
+                    null /* parentId */,
+                    0 /* timestampMillis */,
+                    DataOperations.APPEND,
+                    null /* summary */,
+                    1 /* schemaId */,
+                    "ml.avro" /* manifestList */,
+                    null /* rootManifest */,
+                    1024L /* rootSizeInBytes */,
+                    10L /* firstRowId */,
+                    5L /* addedRows */,
+                    "key-1" /* keyId */))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Root manifest size is not allowed if root manifest is 
null");
+  }
+
+  @TestTemplate
+  void rejectsNegativeRootSizeInBytes() {

Review Comment:
   maybe combine with the previous test using parameterization to reduce 
duplication. this works well if an earlier comment on Preconditions 
consolidation is adopted



##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -241,8 +241,14 @@ private static List<Snapshot> updatePathInSnapshots(
       TableMetadata metadata, String sourcePrefix, String targetPrefix) {
     List<Snapshot> newSnapshots = 
Lists.newArrayListWithCapacity(metadata.snapshots().size());
     for (Snapshot snapshot : metadata.snapshots()) {
+      if (snapshot.rootManifestLocation() != null) {

Review Comment:
   Agreed. Supporting this later needs a change in 
`RewriteTablePathSparkAction` ordering: today the metadata JSON files are 
written first and the rest goes bottom-up (position deletes -> manifests -> 
manifest lists, via `rewrittenManifestLengths`), which only works because v1-v3 
snapshots don't record the manifest list size. For v4, root manifests need to 
be rewritten before the metadata JSON files, with their new lengths passed to 
`replacePaths`. Incremental runs (`startVersion`) also need the rewritten root 
lengths of older snapshots from a previous run, which aren't available today.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to