Hi Jack, Thank you. We've been indeed busy with building the Iceberg data encryption code, since we have quite a demand for this functionality (with timeline requirements..). I've published an initial end-to-end implementation (PR 3053), comprised of a new code that handles the generation of data keys, and of the existing code (with some modifications) from the current PRs listed below (so this is a joint work, with contributions from both of us; I'm sure there are ways to recognize PR co-authorship :).
As I mentioned, this is the simplest version (without double wrapping, column-specific master keys and two-tier key management). I got a prototype for these advanced data encryption features, but thought it might be best to start with an MVP - easier to digest by the community, and allows for a gradual layer-by-layer implementation. In my understanding, MVP can start without key rotation - because the latter has two parts, with the main one (key rotation in KMS) being totally transparent to Iceberg; the other part (re-wrapping of key_metadata and re-writing of manifest files and manifest lists) is required in threat models that cover a risk of master keys being compromised/leaked - so this is a less universal requirement and can be added post-MVP. But if you hold a different view on this, or need the second part of key rotation now, I'm sure this is doable; I just hope it won't slow down the MVP work. Having said that - there is a feature I believe would be a really good addition to the MVP. This is the encryption of manifests and manifest lists. I presume you refer to it in your mail. If you have an internal branch with its implementation - porting this to open source will be much appreciated. We need this capability (yes, the data is encrypted; but the stats are not.. which is not great, even if they actually are highly aggregated, a sort of a range mask). We can chat about this at the upcoming sync, but I support the suggestion to set up a more detailed discussion to align the encryption-related efforts. Cheers, Gidon On Sun, Aug 29, 2021 at 11:08 PM Jack Ye <yezhao...@gmail.com> wrote: > Hi Gidon and Huaxin, > > Thanks for continuing with the effort in Iceberg encryption support. I did > not get enough time to work on this area since the design discussion, so > far I only managed to add key metadata for manifest file, and there are > quite a few changes in our internal branch that I need to port to open > source. I will start to do it in the next few days. > > Regarding the design, I wonder if we should first start with defining the > actions API with a Spark implementation for file encryption key rotation, > and then discuss the user experience. > > In the original design document, I think we did not reach a consensus with > the community around the actual way to expose key rotation functionalities. > In Spark, we can either do it through DDL extension, or implement it as a > procedure. Given that this is a long-running distributed procedure, my > feeling is that the community will lean towards a procedure call. > > We can continue with the discussion around this while first doing the > detailed implementation. Let's set up a discussion around this so that we > can align the efforts. > > Best, > Jack Ye > > > On Wed, Aug 25, 2021 at 4:19 AM Gidon Gershinsky <gg5...@gmail.com> wrote: > >> Hi all, >> >> We have briefly discussed this subject in a June sync, with a decision to >> continue via the mailing list. >> There are a number of pull requests from Jack and myself that implement a >> set of disjoint elements from the high-level design >> <https://docs.google.com/document/d/1kkcjr9KrlB9QagRX3ToulG_Rf-65NMSlVANheDNzJq4/edit?usp=sharing>. >> Some low-level details, such as generation and propagation of data keys, >> are not covered in this document. >> I have created a short (and hopefully simple) doc >> >> https://docs.google.com/document/d/19O_qiQumz_66CdWLpw38GFJEsUpnNxXckP9rnYIQnCo/edit?usp=sharing >> that focuses on these details and describes the bottom-up approach to >> generation of data keys, encryption of data/delete files, and >> options/phases for optimization of key management. The scope of the >> document is intentionally narrow, and currently focuses on the minimal >> simplest option. Reviews are very welcome. Later, this doc will be merged >> in (or referenced from) the master design document. >> >> A PR with a basic encryption DDL has been sent recently by Huaxin, you >> can find it here <https://github.com/apache/iceberg/pull/3013>. Next >> week, I'll send a pull request with an implementation of the minimal >> encryption option. This pull request collects the basics from my PRs 2639, >> 2638, 2640 and Jack's PR 2443; adding the key generation and other code >> that creates an end-to-end implementation of the minimal design >> <https://docs.google.com/document/d/19O_qiQumz_66CdWLpw38GFJEsUpnNxXckP9rnYIQnCo/edit?usp=sharing>. >> This PR comes with an example proposed by Ryan - using a table encryption >> key from a keyfile ("pkcs12" format - the closest thing to the "pem" format >> for symmetric keys). >> Besides the minimal version, I have a draft implementation of more >> advanced data encryption options (including per-column keys, double >> wrapping and two-tier management - all described in the master design doc) >> - but let's take this one step at a time, starting with the simplest option. >> >> Cheers, Gidon >> >
