Hey All, Just bumping this thread. I think this all boils down to whether a) raw key-metadata in table statistics is an existing functionality and key-id is another, maybe better way to do the same, or b) raw key-metadata is not safe in table statistics that we should remove and key-id is it's secure alternative.
I think it's b) but open to hear other opinions. For more details, see my previous mail on this thread. Thanks, Gabor Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. aug. 14., P, 23:14): > Thanks for the answers so far! > > Ryan, for your comments: > *1) Re-grouping the versions* > I can revert the rearrangement of versions, however, this is intentional > from my side. Since all v1-v3 versions are the same, initially I wanted to > merge all three into one. Then I figured we might want to deprecate (and > remove) 'key-metadata' earlier in v3 hence that version is kept separately. > This brings to 2) below: > > *2) Allow using key-metadata field* > 'key-metadata' field for table stats is in the spec, however, we don't > write this field in the Java reference implementation ATM. I'm wondering if > we can expect any custom implementations that support this field. I hope > there aren't many, because based on what I learned from recent > conversations, keeping raw encryption key metadata here is not safe, or > maybe we can consider a security bug, because then the encryption key > metadata could be leaking through the table metadata (that is not encrypted > itself). Let me know if I get this part wrong. > I think the essential questions here: a) Do we expect existing > 'key-metadata' written with this structure? b) Do we consider the > introduction of 'key-id' an enhancement or a bug fix that replaces the > insecure 'key-metadata'? > I'm not sure about a) but for b) I'm leaning towards fixing a security bug > where the 'key-metadata' field should be gone asap. > > *3) Per-file or shared encryption key for stats files* > Using the keys table could serve both approaches, and this is a similar > mechanism to how we use the same for manifest list encryption keys. I'd > leave this flexibility to the writers. > > Thanks, > Gabor > > > > Ryan Blue <[email protected]> ezt írta (időpont: 2026. aug. 11., K, 22:48): > >> -0 >> >> This PR includes changes to add a v4 column for statistics files, but it >> does this in an odd way. First, as Steven pointed out, it changes from >> grouping v2 and v3 together to grouping v1 and v2 together. All 3 versions >> appear to be identical. >> >> Next, the intent of adding the v4 column is to disallow the optional >> `key-metadata` field, but that would mean that statistics files from v3 >> cannot be carried forward into a v4 table without relocating the key to the >> table of encryption keys. That's doable, but it doesn't seem necessary to >> combine updating key management with updating the table spec version. Why >> not allow existing keys to reside where they are but deprecate the field? >> When writing this metadata as v4, I doubt that the upgrade will change the >> encryption scheme, so whatever is translating the metadata would still >> store the key from the statistics table unencrypted, just in a different >> place. >> >> I think that there is also a mismatch between the intent of the keys >> table and how key-metadata is used for stats files today. Right now, there >> is a key-metadata per stats file, but the keys table is intended to have >> one or two keys that are reused, not a key per encrypted file referenced >> from the table metadata. >> >> Rather than require per-file keys to be moved, I think we should just let >> the existing per-file keys remain where they are and add an optional key-id >> for a better way of managing the keys instead. >> >> On Tue, Aug 11, 2026 at 11:31 AM Steven Wu <[email protected]> wrote: >> >>> +1 (binding) >>> >>> On Tue, Aug 11, 2026 at 11:25 AM Alexander Bailey <[email protected]> >>> wrote: >>> >>>> +1 >>>> >>>> Thanks, >>>> Xander >>>> >>>> On Tue, 11 Aug 2026 at 19:01, Anurag Mantripragada < >>>> [email protected]> wrote: >>>> >>>>> +1 (non-binding) >>>>> >>>>> On Tue, Aug 11, 2026 at 8:57 AM Russell Spitzer < >>>>> [email protected]> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> On Tue, Aug 11, 2026 at 1:42 AM Gidon Gershinsky <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> Cheers, Gidon >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 10, 2026 at 10:36 PM Gábor Kaszab < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> The current spec says that the encryption key for table statistics >>>>>>>> should be stored as raw key-metadata. However, since the table >>>>>>>> statistics >>>>>>>> metadata is stored within the unencrypted table metadata, it's not >>>>>>>> secure >>>>>>>> to follow the spec here. >>>>>>>> >>>>>>>> The proposal is two fold: >>>>>>>> 1) Deprecate the 'key-metadata' field in table statistics (Note, >>>>>>>> partition statistics doesn't have this field in the spec) >>>>>>>> 2) Add encryption 'key-id' field to table and partition statistics. >>>>>>>> This points to an encrypted encryption key stored in Table >>>>>>>> metadata's 'encryption-keys' (that in turn points to the KEK in the >>>>>>>> same >>>>>>>> list). This behaviour is similar to how we do the same for manifest >>>>>>>> list >>>>>>>> encryption keys. >>>>>>>> >>>>>>>> PR: https://github.com/apache/iceberg/pull/17533 >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Gabor Kaszab >>>>>>>> >>>>>>>
