Bumping this thread, per the suggestion on the PR I’ve refactored this into a new encryption spec document which encapsulates the StandardKeyMetadata and encryption key structure.
Interested in people’s thoughts here. Best, Xander On Fri, 22 May 2026 at 13:12, Szita Ádám <[email protected]> wrote: > Hi Xander, > > +1, I agree, this needs a more strict specification, thank you for taking > this up, I'll also take a look on your PR ( > https://github.com/apache/iceberg/pull/16527) shortly. > > Cheers, > Adam > > > On Thu, 21 May 2026 at 02:25, Yufei Gu <[email protected]> wrote: > >> Hi Xander, >> >> Thanks for digging into this and documenting the current behavior so >> clearly. >> >> +1 on putting these formats into the spec. At least from an >> interoperability perspective, the current situation creates a practical gap >> between "spec compliant" and "cross-implementation compatible." >> >> Yufei >> >> >> On Wed, May 20, 2026 at 3:14 PM Alexander Bailey <[email protected]> >> wrote: >> >>> Hi all, >>> >>> While implementing table encryption in iceberg-rust, we found a couple >>> of undocumented formats that are required for interoperability but are >>> described in the spec only as "implementation-specific." We >>> have reverse-engineered these from Java's implementation to achieve >>> byte-compatibility. Any future implementation (PyIceberg, etc.) would need >>> to do the same. >>> >>> I'd like to propose that we specify the following in the spec, likely as >>> a new appendix or an expansion of the encryption section. >>> >>> 1. StandardKeyMetadata — the file-level key metadata format >>> >>> The `key_metadata` binary field (field 131 in data files, field 519 in >>> manifest lists) uses a versioned Avro encoding in Java's >>> `StandardKeyMetadata`: >>> >>> Wire format: `[version: 1 byte (0x01)] [Avro binary datum]` >>> >>> V1 schema: >>> ``` >>> required(0, "encryption_key", binary) -- plaintext DEK >>> optional(1, "aad_prefix", binary) -- per-file AAD prefix for AES-GCM >>> optional(2, "file_length", long) -- encrypted file size (for streaming >>> decryption) >>> ``` >>> >>> 2. The encryption-keys list — KEKs vs wrapped DEKs >>> >>> The table-level `encryption-keys` list stores two kinds of entries, >>> distinguished by what `encrypted-by-id` points to: >>> >>> **KEK entries** (`encrypted-by-id` = table master key ID): >>> - `encrypted-key-metadata`: the KEK wrapped by the KMS (opaque, >>> KMS-specific format) >>> - `properties`: includes `"key-timestamp"` (epoch millis) for expiration >>> >>> **Wrapped manifest-list DEK entries** (`encrypted-by-id` = a KEK's >>> key-id): >>> - `encrypted-key-metadata`: the `StandardKeyMetadata` Avro bytes (from >>> #1 above) encrypted with AES-GCM using the referenced KEK, with the KEK's >>> timestamp string as AAD >>> - `properties`: empty >>> >>> The convention for distinguishing these two types of entries, and the >>> wrapping scheme (AES-GCM with the KEK timestamp as AAD to prevent >>> tampering), are not documented anywhere in the spec from what I can see. >>> >>> 3. What can stay "implementation-specific" >>> >>> The KEK's `encrypted-key-metadata` is intentionally opaque, it's >>> whatever the KMS returns from `wrapKey`. That's fine to leave unspecified >>> since it's between the implementation and its KMS provider. >>> >>> ### Why this matters >>> >>> Without specifying #1 and #2, "implementation-specific" becomes a >>> practical interop barrier: tables encrypted by one implementation would be >>> unreadable by another despite both being spec-compliant. These formats are >>> already versioned and frozen in Java - the spec would just be documenting >>> existing reality. >>> >>> Would there be interest in a PR for this? Happy to draft it. >>> >>> Thanks, >>> Xander >>> >>
