Hi Gabor, I agree that in V4, the DEK metadata is best moved into per-file structures (such as snapshots for manifest list files). This will make it easier to manage/clean-up these keys. The `key-d` field indeed becomes obsolete then.
The key encryption keys would stay as they are in the shared `encryption keys` list in TableMetadata, since these keys are shared across the table. Technically, it is possible to re-use DEKs for multiple files, but this requires careful management to prevent breaking the AES GCM cipher. The simple and practical solution is to generate a random DEK per file. Moreover, Iceberg key metadata includes a unique file ID and, in some cases, the file length - so even if a key can be reused, the key metadata cannot. (as for the other technical details - the nonces/IVs are not visible at this level, they are an internal part of PME <https://parquet.apache.org/docs/file-format/data-pages/encryption/> and GCM Stream <https://iceberg.apache.org/gcm-stream-spec/> mechanisms) Cheers, Gidon On Wed, Sep 23, 2026 at 3:18 PM Gábor Kaszab <[email protected]> wrote: > Hey Iceberg Community, > > > > I've been recently involved in conversations around encryption keys and > the current design of how we encrypt manifest list files with the purpose > of extending the design to additional file types like statistics files and > V4 root manifest files. > > I think there are some assumptions with the current design that we should > revisit now. > > > > *Context* > > 1. Manifest list file encryption > > The `encryption-keys` list in table metadata (spec > <https://iceberg.apache.org/spec/#table-metadata-fields>) contains: > > - Encryption keys used for encrypting manifest list files (DEK) > - Key encryption keys (KEK) used for encrypting DEKs. > > The format for them is described here > <https://iceberg.apache.org/spec/#encryption-keys>. Each of them has a > `key-id` and for manifest list files we store a `key-id` in the snapshot to > refer to a DEK in the list. The DEK in turn refers to the KEK used for > encrypting that particular DEK. > > 2. Manifest, data and delete file encryption > > The raw encryption keys are stored directly for these files as > `key-metadata`. See this <https://iceberg.apache.org/spec/#manifest-lists> > or this <https://iceberg.apache.org/spec/#data-file-fields>. > > > > We'll focus on 1) now. > > > > *Assumption* > > The assumption is that a particular DEK in the list can be reused for > encrypting multiple files (manifest list files now, other files also later > on) reducing the space required for storing the DEKs. > > > > *Reality check* > > While I'm not an expert of this area, I did some research, and I think > while it's theoretically feasible, in practice it's overly complicated to > implement such a DEK sharing approach. Here is what I found: > > - It's not just a DEK what we need for encryption but other auxiliary, > generated information like AAD prefix, nonce and other auxiliary > information baked into `key-metadata` > - It's a cryptographical requirement that if DEK is reused then the > some of the generated auxiliary information MUST differ. More particular: > - Some source says reusing the same DEK + nonce pair for multiple > files is "Catastrophic" Link > <https://neilmadden.blog/2024/05/23/galois-counter-mode-and-random-nonces/> > - Even if nonce is generated, there is a theoretical chance of reusing > the same for the same DEK > - Implementation-wise a tracking information is required to keep which > nones are used for each DEK. This information have to be kept in a > persisted way that we don't loose after a restart. Multi-writer scenarios > would make keeping this information even harder > - It's common practice to use a different DEK for different files > > > > Based on the above, I'd be against reusing such keys across multiple files. > > > > *Way forward* > > For V3 manifest lists we already use the approach with a `key-id` in > snapshot referring to an encrypted key metadata in `encryption-keys` list. > This has been out there and I don’t think we can change this now. > > > > For future file types, like V4 root manifests and statistics files, I > think we can consider using a different approach. If we conclude on not > reusing key metadata across files, then I don’t see any point of having the > indirection of storing a `key-id` that refers to an item in the list. > > Instead we can follow a more direct approach, and simply store the > encrypted key metadata of such a file instead of a `key-id`. For instance > for V4 root manifest files we can store `encryption-key` (spec > <https://iceberg.apache.org/spec/#encryption-keys>) directly instead of > `key-id`. > > > Note, this is the structure of an `encryption-key`: > > - Key-id > - Encrypted-key-metadata > - Encrypted-by-id > - Properties > > In this design `key-id` in the structure is unnecessary. (might be > optional?) > > > We could still keep the KEKs in the `encryption-keys` list. > > > What do you think? > > Gabor >
