Hey Iceberg Community,
I've been recently involved in conversations around encryption keys and the current design of how we encrypt manifest list files with the purpose of extending the design to additional file types like statistics files and V4 root manifest files. I think there are some assumptions with the current design that we should revisit now. *Context* 1. Manifest list file encryption The `encryption-keys` list in table metadata (spec <https://iceberg.apache.org/spec/#table-metadata-fields>) contains: - Encryption keys used for encrypting manifest list files (DEK) - Key encryption keys (KEK) used for encrypting DEKs. The format for them is described here <https://iceberg.apache.org/spec/#encryption-keys>. Each of them has a `key-id` and for manifest list files we store a `key-id` in the snapshot to refer to a DEK in the list. The DEK in turn refers to the KEK used for encrypting that particular DEK. 2. Manifest, data and delete file encryption The raw encryption keys are stored directly for these files as `key-metadata`. See this <https://iceberg.apache.org/spec/#manifest-lists> or this <https://iceberg.apache.org/spec/#data-file-fields>. We'll focus on 1) now. *Assumption* The assumption is that a particular DEK in the list can be reused for encrypting multiple files (manifest list files now, other files also later on) reducing the space required for storing the DEKs. *Reality check* While I'm not an expert of this area, I did some research, and I think while it's theoretically feasible, in practice it's overly complicated to implement such a DEK sharing approach. Here is what I found: - It's not just a DEK what we need for encryption but other auxiliary, generated information like AAD prefix, nonce and other auxiliary information baked into `key-metadata` - It's a cryptographical requirement that if DEK is reused then the some of the generated auxiliary information MUST differ. More particular: - Some source says reusing the same DEK + nonce pair for multiple files is "Catastrophic" Link <https://neilmadden.blog/2024/05/23/galois-counter-mode-and-random-nonces/> - Even if nonce is generated, there is a theoretical chance of reusing the same for the same DEK - Implementation-wise a tracking information is required to keep which nones are used for each DEK. This information have to be kept in a persisted way that we don't loose after a restart. Multi-writer scenarios would make keeping this information even harder - It's common practice to use a different DEK for different files Based on the above, I'd be against reusing such keys across multiple files. *Way forward* For V3 manifest lists we already use the approach with a `key-id` in snapshot referring to an encrypted key metadata in `encryption-keys` list. This has been out there and I don’t think we can change this now. For future file types, like V4 root manifests and statistics files, I think we can consider using a different approach. If we conclude on not reusing key metadata across files, then I don’t see any point of having the indirection of storing a `key-id` that refers to an item in the list. Instead we can follow a more direct approach, and simply store the encrypted key metadata of such a file instead of a `key-id`. For instance for V4 root manifest files we can store `encryption-key` (spec <https://iceberg.apache.org/spec/#encryption-keys>) directly instead of `key-id`. Note, this is the structure of an `encryption-key`: - Key-id - Encrypted-key-metadata - Encrypted-by-id - Properties In this design `key-id` in the structure is unnecessary. (might be optional?) We could still keep the KEKs in the `encryption-keys` list. What do you think? Gabor
