Hi, Just to clarify: PF~ allows older readers to read data as long as they only try to access unencrypted columns. What happens when older readers do try to access encrypted columns?
Also, by older readers do you specificially mean the current Java library or all existing language bindings? Thanks, Zoltan On Tue, Sep 18, 2018 at 9:45 AM Gidon Gershinsky <[email protected]> wrote: > Hi all, > > This week, 8 months after the first call for goals feedback and > requirements :), I got a new one - enabling old Parquet readers to access > data of unencrypted columns in encrypted files. > Better late than never.. But actually it doesn't sound unreasonable, and > deserved at least a consideration. > > Let me describe the options (the way I see them). Any community feedback is > welcome. > > But first, a little tech intro. Encrypted Parquet files can be created in > two modes - with an encrypted footer (lets call this an 'EF' mode for the > purpose of this discussion), or with a plaintext footer ('PF' mode). > EF is significantly more secure - it protects all data and metadata in a > file, including the schema, number of rows, key-value properties, column > names, column sort order, list of encrypted columns and metadata of the > column encryption keys. > PF hides the data, but leaks all of these metadata fields. Moreover, EF > makes the footer tamper-proof, while PF doesn't. > The reason we have the PF option is to let users with relaxed security > requirements to enable readers, that don't have access to any keys, to read > unencrypted columns in a file. > > For encrypted columns, both EH and PH hide the ColumnMetaData - including > the min/max stats, number of values, data offset, data size and other > fields. Old Parquet readers obviously can't read EF files. But they can't > also read PF files - because old readers need access to data offset and > size of every column in a file, event if they try to read just one column > (this is fixed in an encryption pull request). > > Now, the options: > > 1) Don't allow old Parquet readers to read encrypted files. Organizations > that start working with encrypted data, will update their analytic > frameworks to use an encrypting Parquet version. This includes both > frameworks that write/read encrypted columns, and frameworks that work only > with unencrypted columns. The former and latter can technically be the same > framework, just different instances of it. The update can be done in one of > the following ways: > a. Upgrade Parquet version to the latest one, supporting encryption. This > might require some changes in framework code, unrelated to encryption. > b. Use the original old Parquet version, with an added encryption support > (requires rebuilding the framework, no code changes). This is not hard, I'm > doing it for Parquet 1.8.2 in order to build and run Spark 2.3.0 with > encrypted data. > I think I can post this for 1.8.2 and other versions, with some help from > the community. > > 2) Replace PF with PF~, in order to allow old Parquet readers to read > unencrypted columns in encrypted files. PF~ is a little less secure and a > little less elegant version of PF. Less secure because it has to expose the > offset and size of encrypted column data. But actually its not > catastrophic, and in any case, organizations with higher security > requirements will use the EF mode. Others can start with PF~ for a > transition period, and switch to EF later. > PH~ requires changing 2 lines in the parquet.thrift file, and a few dozen > lines in the implementation. I've played with this today, seems quite > feasible. > So, unless the community strongly favors option 1, I'm inclined to proceed > with 2, should take up to a week to get the prs submitted. > > Cheers, Gidon. >
