gaborkaszab commented on code in PR #18503:
URL: https://github.com/apache/iceberg/pull/18503#discussion_r4586559680


##########
format/spec.md:
##########
@@ -1251,11 +1251,13 @@ A snapshot consists of the following fields:
     | _required_ | **`sequence-number`**        | A monotonically increasing 
long that tracks the order of changes to a table |
     | _required_ | **`timestamp-ms`**           | A timestamp when the 
snapshot was created, used for garbage collection and table inspection |
     | _required_ | **`root-manifest`**          | The location of the root 
manifest for this snapshot |
+    | _required_ | **`root-manifest-size-in-bytes`** | Size of the root 
manifest file in bytes |
     | _required_ | **`summary`**                | A string map that summarizes 
the snapshot changes, including `operation` as a _required_ field (see below) |
     | _optional_ | **`schema-id`**              | ID of the table's current 
schema when the snapshot was created |
     | _required_ | **`first-row-id`**           | The first `_row_id` assigned 
to the first row in the first data file in the first manifest, see [Row 
Lineage](#row-lineage) |
     | _required_ | **`added-rows`**             | The upper bound of the 
number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
-    | _optional_ | **`key-id`**                 | ID of the encryption key 
that encrypts the root manifest key metadata |
+    | _optional_ | **`key-id`**                 | ID of the encryption key 
that encrypts the root manifest |

Review Comment:
   My take is that we shouldn't allow `key-id` in V4 and use the new snapshot 
encryption fields instead. This is exactly how we started this PR, but I wasn't 
sure what the exact spec language is to express this.
   
   Apparently, the `key-id` field isn't any different from how we treat 
`manifest-list` from a V4 reader/writer's perspective: read them as optional, 
don't write them. We simply dropped the `manifest-list` field in the V4 tab, to 
follow the existing pattern I'm dropping `key-id` too.
   
   Splitting such columns into read and write requirements could make sense, we 
also discussed this on one of the V4 syncs. However, I'd do that separately if 
we decide so.
   
   WDYT @rdblue ?



##########
format/spec.md:
##########
@@ -1269,6 +1271,8 @@ The snapshot summary's `operation` field is used by some 
operations, like snapsh
 
 For other optional snapshot summary fields, see [Appendix 
F](#optional-snapshot-summary-fields).
 
+The `encrypted-key-metadata` and `encrypted-by-id` fields must be set together.

Review Comment:
   Keys can't be plaintext here, I think that's what the field name 
`encrypted-key-metadata` and its description suggests.
   
   However, I did some digging and [I see the original intent 
why](https://github.com/apache/iceberg/pull/12162#discussion_r1940256313) 
`encrypted-by-id` is optional: such keys can be encrypted by KEK that we refer 
to by `encrypted-by-id`, or can be encrypted directly by the KMS. In the latter 
case we can leave the field empty.
   
   I'll use your suggestion here, thanks!



##########
format/spec.md:
##########
@@ -1241,7 +1241,7 @@ A snapshot consists of the following fields:
     | _optional_ | _optional_ | _optional_ | **`schema-id`**              | ID 
of the table's current schema when the snapshot was created |
     |            |            | _required_ | **`first-row-id`**           | 
The first `_row_id` assigned to the first row in the first data file in the 
first manifest, see [Row Lineage](#row-lineage) |
     |            |            | _required_ | **`added-rows`**             | 
The upper bound of the number of rows with assigned row IDs, see [Row 
Lineage](#row-lineage) |
-    |            |            | _optional_ | **`key-id`**                 | ID 
of the encryption key that encrypts the manifest list key metadata |
+    |            |            | _optional_ | **`key-id`**                 | ID 
of the encryption key that encrypts the manifest list |

Review Comment:
   Sure. I reverted this and create a separate PR to cover the fix: 
https://github.com/apache/iceberg/pull/18998



##########
format/spec.md:
##########
@@ -2263,6 +2267,10 @@ Snapshot timestamp changes:
 * When committed while the table is at version 4 or higher, a snapshot's 
`timestamp-ms` must be strictly greater than its parent snapshot's 
`timestamp-ms` on the same branch.
 * Snapshots with no parent and snapshots that existed before a table was 
upgraded to v4 are not constrained.
 
+Snapshot encryption key changes:
+
+* Writers may store the root manifest's encryption key metadata directly in 
the snapshot's `encrypted-key-metadata` and `encrypted-by-id` fields

Review Comment:
   Thx, applied



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to