gaborkaszab commented on code in PR #18503:
URL: https://github.com/apache/iceberg/pull/18503#discussion_r4586559680
##########
format/spec.md:
##########
@@ -1251,11 +1251,13 @@ A snapshot consists of the following fields:
| _required_ | **`sequence-number`** | A monotonically increasing
long that tracks the order of changes to a table |
| _required_ | **`timestamp-ms`** | A timestamp when the
snapshot was created, used for garbage collection and table inspection |
| _required_ | **`root-manifest`** | The location of the root
manifest for this snapshot |
+ | _required_ | **`root-manifest-size-in-bytes`** | Size of the root
manifest file in bytes |
| _required_ | **`summary`** | A string map that summarizes
the snapshot changes, including `operation` as a _required_ field (see below) |
| _optional_ | **`schema-id`** | ID of the table's current
schema when the snapshot was created |
| _required_ | **`first-row-id`** | The first `_row_id` assigned
to the first row in the first data file in the first manifest, see [Row
Lineage](#row-lineage) |
| _required_ | **`added-rows`** | The upper bound of the
number of rows with assigned row IDs, see [Row Lineage](#row-lineage) |
- | _optional_ | **`key-id`** | ID of the encryption key
that encrypts the root manifest key metadata |
+ | _optional_ | **`key-id`** | ID of the encryption key
that encrypts the root manifest |
Review Comment:
My take is that we shouldn't allow `key-id` in V4 and use the new snapshot
encryption fields instead. This is exactly how we started this PR, but I wasn't
sure what the exact spec language is to express this.
Apparently, the `key-id` field isn't any different from how we treat
`manifest-list` from a V4 reader/writer's perspective: read them as optional,
don't write them. We simply dropped the `manifest-list` field in the V4 tab, to
follow the existing pattern I'm dropping `key-id` too.
Splitting such columns into read and write requirements could make sense, we
also discussed this on one of the V4 syncs. However, I'd do that separately if
we decide so.
WDYT @rdblue ?
##########
format/spec.md:
##########
@@ -1269,6 +1271,8 @@ The snapshot summary's `operation` field is used by some
operations, like snapsh
For other optional snapshot summary fields, see [Appendix
F](#optional-snapshot-summary-fields).
+The `encrypted-key-metadata` and `encrypted-by-id` fields must be set together.
Review Comment:
Keys can't be plaintext here, I think that's what the field name
`encrypted-key-metadata` and its description suggests.
However, I did some digging and [I see the original intent
why](https://github.com/apache/iceberg/pull/12162#discussion_r1940256313)
`encrypted-by-id` is optional: such keys can be encrypted by KEK that we refer
to by `encrypted-by-id`, or can be encrypted directly by the KMS. In the latter
case we can leave the field empty.
I'll use your suggestion here, thanks!
##########
format/spec.md:
##########
@@ -1241,7 +1241,7 @@ A snapshot consists of the following fields:
| _optional_ | _optional_ | _optional_ | **`schema-id`** | ID
of the table's current schema when the snapshot was created |
| | | _required_ | **`first-row-id`** |
The first `_row_id` assigned to the first row in the first data file in the
first manifest, see [Row Lineage](#row-lineage) |
| | | _required_ | **`added-rows`** |
The upper bound of the number of rows with assigned row IDs, see [Row
Lineage](#row-lineage) |
- | | | _optional_ | **`key-id`** | ID
of the encryption key that encrypts the manifest list key metadata |
+ | | | _optional_ | **`key-id`** | ID
of the encryption key that encrypts the manifest list |
Review Comment:
Sure. I reverted this and create a separate PR to cover the fix:
https://github.com/apache/iceberg/pull/18998
##########
format/spec.md:
##########
@@ -2263,6 +2267,10 @@ Snapshot timestamp changes:
* When committed while the table is at version 4 or higher, a snapshot's
`timestamp-ms` must be strictly greater than its parent snapshot's
`timestamp-ms` on the same branch.
* Snapshots with no parent and snapshots that existed before a table was
upgraded to v4 are not constrained.
+Snapshot encryption key changes:
+
+* Writers may store the root manifest's encryption key metadata directly in
the snapshot's `encrypted-key-metadata` and `encrypted-by-id` fields
Review Comment:
Thx, applied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]