Hi Gidon, Gabor, I agree that we should generate a fresh DEK for each file - in fact I would personally want this to be made explicit. This came up when reviewing https://github.com/apache/iceberg/pull/16353
My initial preference reading this was to retain the existing key-id / encryption-keys pattern so implementations could use the same approach across these file types. The point about the metadata being specific to the file, and simplifying its cleanup, makes sense though. I’d like us to preserve a common encrypted key metadata structure and wrapping/unwrapping model as we make this change. We should be able to share that implementation whether the metadata is resolved through a V3 key ID or obtained directly from a V4 per-file structure. With that, I can see the benefit of keeping the encrypted DEK metadata alongside the file reference and retaining shared KEKs in encryption-keys. The per-file DEK entry would no longer need its own key-id, but would still need to identify the KEK used to wrap it. Thanks, Xander On Wed, 23 Sept 2026 at 14:42, Gidon Gershinsky <[email protected]> wrote: > Hi Gabor, > > I agree that in V4, the DEK metadata is best moved into per-file > structures (such as snapshots for manifest list files). This will make it > easier to manage/clean-up these keys. The `key-d` field indeed becomes > obsolete then. > > The key encryption keys would stay as they are in the shared `encryption > keys` list in TableMetadata, since these keys are shared across the table. > > Technically, it is possible to re-use DEKs for multiple files, but this > requires careful management to prevent breaking the AES GCM cipher. The > simple and practical solution is to generate a random DEK per file. > Moreover, Iceberg key metadata includes a unique file ID and, in some > cases, the file length - so even if a key can be reused, the key metadata > cannot. > > (as for the other technical details - the nonces/IVs are not visible at > this level, they are an internal part of PME > <https://parquet.apache.org/docs/file-format/data-pages/encryption/> and GCM > Stream <https://iceberg.apache.org/gcm-stream-spec/> mechanisms) > > Cheers, Gidon > > > On Wed, Sep 23, 2026 at 3:18 PM Gábor Kaszab <[email protected]> > wrote: > >> Hey Iceberg Community, >> >> >> >> I've been recently involved in conversations around encryption keys and >> the current design of how we encrypt manifest list files with the purpose >> of extending the design to additional file types like statistics files and >> V4 root manifest files. >> >> I think there are some assumptions with the current design that we should >> revisit now. >> >> >> >> *Context* >> >> 1. Manifest list file encryption >> >> The `encryption-keys` list in table metadata (spec >> <https://iceberg.apache.org/spec/#table-metadata-fields>) contains: >> >> - Encryption keys used for encrypting manifest list files (DEK) >> - Key encryption keys (KEK) used for encrypting DEKs. >> >> The format for them is described here >> <https://iceberg.apache.org/spec/#encryption-keys>. Each of them has a >> `key-id` and for manifest list files we store a `key-id` in the snapshot to >> refer to a DEK in the list. The DEK in turn refers to the KEK used for >> encrypting that particular DEK. >> >> 2. Manifest, data and delete file encryption >> >> The raw encryption keys are stored directly for these files as >> `key-metadata`. See this >> <https://iceberg.apache.org/spec/#manifest-lists> or this >> <https://iceberg.apache.org/spec/#data-file-fields>. >> >> >> >> We'll focus on 1) now. >> >> >> >> *Assumption* >> >> The assumption is that a particular DEK in the list can be reused for >> encrypting multiple files (manifest list files now, other files also later >> on) reducing the space required for storing the DEKs. >> >> >> >> *Reality check* >> >> While I'm not an expert of this area, I did some research, and I think >> while it's theoretically feasible, in practice it's overly complicated to >> implement such a DEK sharing approach. Here is what I found: >> >> - It's not just a DEK what we need for encryption but other >> auxiliary, generated information like AAD prefix, nonce and other >> auxiliary >> information baked into `key-metadata` >> - It's a cryptographical requirement that if DEK is reused then the >> some of the generated auxiliary information MUST differ. More particular: >> - Some source says reusing the same DEK + nonce pair for multiple >> files is "Catastrophic" Link >> >> <https://neilmadden.blog/2024/05/23/galois-counter-mode-and-random-nonces/> >> - Even if nonce is generated, there is a theoretical chance of >> reusing the same for the same DEK >> - Implementation-wise a tracking information is required to keep >> which nones are used for each DEK. This information have to be kept in a >> persisted way that we don't loose after a restart. Multi-writer scenarios >> would make keeping this information even harder >> - It's common practice to use a different DEK for different files >> >> >> >> Based on the above, I'd be against reusing such keys across multiple >> files. >> >> >> >> *Way forward* >> >> For V3 manifest lists we already use the approach with a `key-id` in >> snapshot referring to an encrypted key metadata in `encryption-keys` list. >> This has been out there and I don’t think we can change this now. >> >> >> >> For future file types, like V4 root manifests and statistics files, I >> think we can consider using a different approach. If we conclude on not >> reusing key metadata across files, then I don’t see any point of having the >> indirection of storing a `key-id` that refers to an item in the list. >> >> Instead we can follow a more direct approach, and simply store the >> encrypted key metadata of such a file instead of a `key-id`. For instance >> for V4 root manifest files we can store `encryption-key` (spec >> <https://iceberg.apache.org/spec/#encryption-keys>) directly instead of >> `key-id`. >> >> >> Note, this is the structure of an `encryption-key`: >> >> - Key-id >> - Encrypted-key-metadata >> - Encrypted-by-id >> - Properties >> >> In this design `key-id` in the structure is unnecessary. (might be >> optional?) >> >> >> We could still keep the KEKs in the `encryption-keys` list. >> >> >> What do you think? >> >> Gabor >> >
