ggershinsky commented on PR #17417: URL: https://github.com/apache/iceberg/pull/17417#issuecomment-5133962320
Yep, the key metadata keeps the encryption key in the open.. It can be encrypted by a "key encryption key" - like we do for the snapshot (manifest list file) keys. But then we have a storage issue, because we also need a pointer to the "key encryption key". For snapshots, we don't have a`key_metadata` field - instead, we keep a `key-id` that points to the `encrypted-keys` table entry (and there, we have the key encryption key id) A (technically) straightforward solution would be to add a `key-id` field to the `statistics` struct (and deprecate the `key_metadata`field), but I don't know how practical it is wrt the spec process, V4 etc. There might be less straightforward solutions that don't require spec changes, but they either are looking not great (like packing everything needed inside the `key_metadata` in a custom format) or have substantial performance implications (like calling KMS for each file/snapshot, instead of using a cached key encryption key). The latter also has some security issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
