Thanks for sharing your view on this, Gidon! Noted, theoretically there could be implementations that use key-metadata in a safe way, so let's keep it and offer a key-id based mechanism as an alternative.
I updated the PR <https://github.com/apache/iceberg/pull/17533>, let me know if this is what you meant. Let's conclude on the approach there, and I'll open another vote thead after that. Thanks! Gabor Gidon Gershinsky <[email protected]> ezt írta (időpont: 2026. aug. 25., K, 9:13): > I think option (a) is practical. There could be other implementations > where key-metadata does not expose a cleartext key. > Leaving it in place (potentially deprecated), and adding a new key-id > field, should be fine. > That'd solve the problem for the current encryption implementation, while > keeping the tools for other implementations. > > Cheers, Gidon > > > On Mon, Aug 24, 2026 at 5:37 PM Gábor Kaszab <[email protected]> > wrote: > >> Hey All, >> >> Just bumping this thread. >> I think this all boils down to whether a) raw key-metadata in table >> statistics is an existing functionality and key-id is another, maybe better >> way to do the same, or b) raw key-metadata is not safe in table statistics >> that we should remove and key-id is it's secure alternative. >> >> I think it's b) but open to hear other opinions. For more details, see my >> previous mail on this thread. >> >> Thanks, >> Gabor >> >> Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. aug. 14., >> P, 23:14): >> >>> Thanks for the answers so far! >>> >>> Ryan, for your comments: >>> *1) Re-grouping the versions* >>> I can revert the rearrangement of versions, however, this is intentional >>> from my side. Since all v1-v3 versions are the same, initially I wanted to >>> merge all three into one. Then I figured we might want to deprecate (and >>> remove) 'key-metadata' earlier in v3 hence that version is kept separately. >>> This brings to 2) below: >>> >>> *2) Allow using key-metadata field* >>> 'key-metadata' field for table stats is in the spec, however, we don't >>> write this field in the Java reference implementation ATM. I'm wondering if >>> we can expect any custom implementations that support this field. I hope >>> there aren't many, because based on what I learned from recent >>> conversations, keeping raw encryption key metadata here is not safe, or >>> maybe we can consider a security bug, because then the encryption key >>> metadata could be leaking through the table metadata (that is not encrypted >>> itself). Let me know if I get this part wrong. >>> I think the essential questions here: a) Do we expect existing >>> 'key-metadata' written with this structure? b) Do we consider the >>> introduction of 'key-id' an enhancement or a bug fix that replaces the >>> insecure 'key-metadata'? >>> I'm not sure about a) but for b) I'm leaning towards fixing a security >>> bug where the 'key-metadata' field should be gone asap. >>> >>> *3) Per-file or shared encryption key for stats files* >>> Using the keys table could serve both approaches, and this is a similar >>> mechanism to how we use the same for manifest list encryption keys. I'd >>> leave this flexibility to the writers. >>> >>> Thanks, >>> Gabor >>> >>> >>> >>> Ryan Blue <[email protected]> ezt írta (időpont: 2026. aug. 11., K, >>> 22:48): >>> >>>> -0 >>>> >>>> This PR includes changes to add a v4 column for statistics files, but >>>> it does this in an odd way. First, as Steven pointed out, it changes from >>>> grouping v2 and v3 together to grouping v1 and v2 together. All 3 versions >>>> appear to be identical. >>>> >>>> Next, the intent of adding the v4 column is to disallow the optional >>>> `key-metadata` field, but that would mean that statistics files from v3 >>>> cannot be carried forward into a v4 table without relocating the key to the >>>> table of encryption keys. That's doable, but it doesn't seem necessary to >>>> combine updating key management with updating the table spec version. Why >>>> not allow existing keys to reside where they are but deprecate the field? >>>> When writing this metadata as v4, I doubt that the upgrade will change the >>>> encryption scheme, so whatever is translating the metadata would still >>>> store the key from the statistics table unencrypted, just in a different >>>> place. >>>> >>>> I think that there is also a mismatch between the intent of the keys >>>> table and how key-metadata is used for stats files today. Right now, there >>>> is a key-metadata per stats file, but the keys table is intended to have >>>> one or two keys that are reused, not a key per encrypted file referenced >>>> from the table metadata. >>>> >>>> Rather than require per-file keys to be moved, I think we should just >>>> let the existing per-file keys remain where they are and add an optional >>>> key-id for a better way of managing the keys instead. >>>> >>>> On Tue, Aug 11, 2026 at 11:31 AM Steven Wu <[email protected]> >>>> wrote: >>>> >>>>> +1 (binding) >>>>> >>>>> On Tue, Aug 11, 2026 at 11:25 AM Alexander Bailey <[email protected]> >>>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> Thanks, >>>>>> Xander >>>>>> >>>>>> On Tue, 11 Aug 2026 at 19:01, Anurag Mantripragada < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +1 (non-binding) >>>>>>> >>>>>>> On Tue, Aug 11, 2026 at 8:57 AM Russell Spitzer < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> On Tue, Aug 11, 2026 at 1:42 AM Gidon Gershinsky <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> Cheers, Gidon >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 10, 2026 at 10:36 PM Gábor Kaszab < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> The current spec says that the encryption key for table >>>>>>>>>> statistics should be stored as raw key-metadata. However, since the >>>>>>>>>> table >>>>>>>>>> statistics metadata is stored within the unencrypted table metadata, >>>>>>>>>> it's >>>>>>>>>> not secure to follow the spec here. >>>>>>>>>> >>>>>>>>>> The proposal is two fold: >>>>>>>>>> 1) Deprecate the 'key-metadata' field in table statistics (Note, >>>>>>>>>> partition statistics doesn't have this field in the spec) >>>>>>>>>> 2) Add encryption 'key-id' field to table and partition >>>>>>>>>> statistics. This points to an encrypted encryption key stored in >>>>>>>>>> Table >>>>>>>>>> metadata's 'encryption-keys' (that in turn points to the KEK in the >>>>>>>>>> same >>>>>>>>>> list). This behaviour is similar to how we do the same for manifest >>>>>>>>>> list >>>>>>>>>> encryption keys. >>>>>>>>>> >>>>>>>>>> PR: https://github.com/apache/iceberg/pull/17533 >>>>>>>>>> >>>>>>>>>> Best Regards, >>>>>>>>>> Gabor Kaszab >>>>>>>>>> >>>>>>>>>
