Thanks Gidon for sending out! Here is the link for the Schema-driven design doc <https://docs.google.com/document/d/17GTQAezl1ZC1pMNHjYU_bPVxMU6DIPjtXOiLclXUlyA> .
On Mon, May 11, 2020 at 6:14 AM Gidon Gershinsky <[email protected]> wrote: > Hi all, > > Now that the low level interface to Parquet encryption is merged in > parquet-cpp, and close to completion in parquet-mr, we need to get back to > the subject of a high level interface, that allows to use Parquet > encryption in a simple, almost transparent way; and helps with management > of encryption keys. > > What has changed in this field since June'19, when we have last > discussed it? > - the basic Parquet encryption layer and its low level interface are mostly > complete > - the two alternatives to high level interfaces we had (properties-driven, > and schema-driven), are not mutually exclusive anymore. Together with > Xinli, Gabor and Maya, we have managed to create a simple Crypto Factory > interface mechanism (PARQUET-1817 > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_PARQUET-2D1817&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=HgtpTTM_Ioa61sMBfZKozWM5tNOs8ZanDhyb9mwcOtA&e= > >, already merged in > parquet-mr/encryption), that allows to plug in any of the two alternatives > - or any other implementation of a high level encryption interface. > - the properties-driven interface, and the key management tools used for > its implementation, have matured significantly, and are already deployed in > production. > - I presume the schema-driven interface (crypto-interface with schema > activation) has significantly matured as well. > > The draft design of the Properties-driven encryption is here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1boH6HPkG0ZhgxcaRkGk3QpZ8X-5FJ91uXZwVGwYN45St4_edit-3Fusp-3Dsharing&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=ZRk5BcX4fZtHBdesaR1Mb1-0k0FAgrIYas6dGwazcqE&e= > > - Key management tools (leveraged to build the properties-driven > encryption, but have a wider applicability), design: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_1bEu903840yb95k9q2X-2DBlsYKuXoygE4VnMDl9xz-5Fzhk_edit-3Fusp-3Dsharing&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=dOqanWhKkhBmprgIE_qO1rbEq08dAB60PyFJh3dvUYo&e= > > - Code: the draft pull request that implements Properties-driven > encryption (and Key management tools) is here: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_pull_615&d=DwIBaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=wr291R18SLXzPPNRFVMr0TsZJaFMInOCSszCLCL-Uk0&s=CljrTqJHk--geQJLa18DN6iKuPcKH-_J2Lsneowqmck&e= > > > Xinli informs that the Schema-driven design doc is ready too, and a link > will be sent soon. > > > All feedback from the community will be appreciated. > > Cheers, Gidon. > -- Xinli Shang
