AFAIK encryption is not fully supported in parquet-cpp 1.5.0 so reading and writing files with encryption is not possible. Am I wrong about that?
I'm wholly supportive of changing the project to use a released version of the Parquet format, so let's do that ASAP. On Tue, Sep 25, 2018 at 1:30 PM Gidon Gershinsky <[email protected]> wrote: > > Yep! (sent in parallel :) > > On Tue, Sep 25, 2018 at 8:19 PM Zoltan Ivanfi <[email protected]> > wrote: > > > Hi, > > > > As a short update, I just checked the PR for PARQUET-1419 and although in > > its current form it is a breaking change, it can be easily rewritten to > > become backwards-compatible so this part of the problem does not apply any > > more. > > > > Br, > > > > Zoltan > > > > On Tue, Sep 25, 2018 at 7:10 PM Zoltan Ivanfi <[email protected]> wrote: > > > > > Hi, > > > > > > On the Parquet sync we discussed that the practice of maintaining a copy > > > of parquet.thrift in parquet-cpp is dangerous and that we must take care > > to > > > not release parquet-format changes in parquet-cpp before we officially > > > release them in parquet-format. As I got back to my computer and started > > to > > > create a JIRA about this, I noticed that unfortunately this has already > > > happened. > > > > > > The encryption-releated parquet.thrift changes have not only been added > > to > > > only parquet-format, but to parquet-cpp as well, and these changes got > > > released in parquet-cpp 1.5.0. This is very unfortunate, because > > > PARQUET-1419 would change the encryption in a breaking way, which is only > > > acceptable as long as the original is not released. Additionally, it has > > > been discussed that a formal voting should take place before > > incorporating > > > the encryption features in the format. > > > > > > Now that parquet-cpp has already shipped these changes, we must choose > > the > > > lesser evil of the following two options: > > > > > > - Release a parquet-cpp 1.6.0 with a breaking change and risk that > > > encrypted data files already written with parquet-cpp 1.5.0 will not > > be > > > readable any more. > > > - Release the encryption in parquet-format as it is, regardless of > > > voting results and discard PARQUET-1419. > > > > > > Personally I have a hard time deciding which one I consider lesser evil. > > > What are your opinions? > > > > > > Thanks, > > > > > > Zoltan > > > > >
