I think option (a) is practical. There could be other implementations where key-metadata does not expose a cleartext key. Leaving it in place (potentially deprecated), and adding a new key-id field, should be fine. That'd solve the problem for the current encryption implementation, while keeping the tools for other implementations.
Cheers, Gidon On Mon, Aug 24, 2026 at 5:37 PM Gábor Kaszab <[email protected]> wrote: > Hey All, > > Just bumping this thread. > I think this all boils down to whether a) raw key-metadata in table > statistics is an existing functionality and key-id is another, maybe better > way to do the same, or b) raw key-metadata is not safe in table statistics > that we should remove and key-id is it's secure alternative. > > I think it's b) but open to hear other opinions. For more details, see my > previous mail on this thread. > > Thanks, > Gabor > > Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. aug. 14., > P, 23:14): > >> Thanks for the answers so far! >> >> Ryan, for your comments: >> *1) Re-grouping the versions* >> I can revert the rearrangement of versions, however, this is intentional >> from my side. Since all v1-v3 versions are the same, initially I wanted to >> merge all three into one. Then I figured we might want to deprecate (and >> remove) 'key-metadata' earlier in v3 hence that version is kept separately. >> This brings to 2) below: >> >> *2) Allow using key-metadata field* >> 'key-metadata' field for table stats is in the spec, however, we don't >> write this field in the Java reference implementation ATM. I'm wondering if >> we can expect any custom implementations that support this field. I hope >> there aren't many, because based on what I learned from recent >> conversations, keeping raw encryption key metadata here is not safe, or >> maybe we can consider a security bug, because then the encryption key >> metadata could be leaking through the table metadata (that is not encrypted >> itself). Let me know if I get this part wrong. >> I think the essential questions here: a) Do we expect existing >> 'key-metadata' written with this structure? b) Do we consider the >> introduction of 'key-id' an enhancement or a bug fix that replaces the >> insecure 'key-metadata'? >> I'm not sure about a) but for b) I'm leaning towards fixing a security >> bug where the 'key-metadata' field should be gone asap. >> >> *3) Per-file or shared encryption key for stats files* >> Using the keys table could serve both approaches, and this is a similar >> mechanism to how we use the same for manifest list encryption keys. I'd >> leave this flexibility to the writers. >> >> Thanks, >> Gabor >> >> >> >> Ryan Blue <[email protected]> ezt írta (időpont: 2026. aug. 11., K, 22:48): >> >>> -0 >>> >>> This PR includes changes to add a v4 column for statistics files, but it >>> does this in an odd way. First, as Steven pointed out, it changes from >>> grouping v2 and v3 together to grouping v1 and v2 together. All 3 versions >>> appear to be identical. >>> >>> Next, the intent of adding the v4 column is to disallow the optional >>> `key-metadata` field, but that would mean that statistics files from v3 >>> cannot be carried forward into a v4 table without relocating the key to the >>> table of encryption keys. That's doable, but it doesn't seem necessary to >>> combine updating key management with updating the table spec version. Why >>> not allow existing keys to reside where they are but deprecate the field? >>> When writing this metadata as v4, I doubt that the upgrade will change the >>> encryption scheme, so whatever is translating the metadata would still >>> store the key from the statistics table unencrypted, just in a different >>> place. >>> >>> I think that there is also a mismatch between the intent of the keys >>> table and how key-metadata is used for stats files today. Right now, there >>> is a key-metadata per stats file, but the keys table is intended to have >>> one or two keys that are reused, not a key per encrypted file referenced >>> from the table metadata. >>> >>> Rather than require per-file keys to be moved, I think we should just >>> let the existing per-file keys remain where they are and add an optional >>> key-id for a better way of managing the keys instead. >>> >>> On Tue, Aug 11, 2026 at 11:31 AM Steven Wu <[email protected]> wrote: >>> >>>> +1 (binding) >>>> >>>> On Tue, Aug 11, 2026 at 11:25 AM Alexander Bailey <[email protected]> >>>> wrote: >>>> >>>>> +1 >>>>> >>>>> Thanks, >>>>> Xander >>>>> >>>>> On Tue, 11 Aug 2026 at 19:01, Anurag Mantripragada < >>>>> [email protected]> wrote: >>>>> >>>>>> +1 (non-binding) >>>>>> >>>>>> On Tue, Aug 11, 2026 at 8:57 AM Russell Spitzer < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> On Tue, Aug 11, 2026 at 1:42 AM Gidon Gershinsky <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> Cheers, Gidon >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 10, 2026 at 10:36 PM Gábor Kaszab < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> The current spec says that the encryption key for table statistics >>>>>>>>> should be stored as raw key-metadata. However, since the table >>>>>>>>> statistics >>>>>>>>> metadata is stored within the unencrypted table metadata, it's not >>>>>>>>> secure >>>>>>>>> to follow the spec here. >>>>>>>>> >>>>>>>>> The proposal is two fold: >>>>>>>>> 1) Deprecate the 'key-metadata' field in table statistics (Note, >>>>>>>>> partition statistics doesn't have this field in the spec) >>>>>>>>> 2) Add encryption 'key-id' field to table and partition >>>>>>>>> statistics. This points to an encrypted encryption key stored in Table >>>>>>>>> metadata's 'encryption-keys' (that in turn points to the KEK in the >>>>>>>>> same >>>>>>>>> list). This behaviour is similar to how we do the same for manifest >>>>>>>>> list >>>>>>>>> encryption keys. >>>>>>>>> >>>>>>>>> PR: https://github.com/apache/iceberg/pull/17533 >>>>>>>>> >>>>>>>>> Best Regards, >>>>>>>>> Gabor Kaszab >>>>>>>>> >>>>>>>>
