To give a bit of input: I have written a simple Parquet file with pyarrow/parquet-cpp, one with Thrift 0.11, the other with Thrift 0.12 and they were identical. We have been using Thrift 0.11 for quite some while in there, so I would assume that there are no problems between the version. Otherwise I would have expected users reporting many issues about pyarrow/Spark interop.
Uwe On Sun, Jan 27, 2019, at 3:33 AM, Wes McKinney wrote: > I thought the Thrift binary protocol is stable at this point (I'm not > sure what process Apache Thrift uses to ensure this), but I agree it > would be worth a smoke test of new-Thrift vs. old-Thrift. I've build > parquet-cpp with 0.12 and not had any issues > > - Wes > > On Fri, Jan 25, 2019 at 10:34 AM Ryan Blue <[email protected]> wrote: > > > > The thrift dependency in parquet-format and the one in parquet-mr can > > coexist because we shade the one in parquet-format. Thrift should also be > > binary compatible, although I don't think they publish any guarantees. > > > > On Fri, Jan 25, 2019 at 12:53 AM Uwe L. Korn <[email protected]> wrote: > > > > > As an FYI: parquet-cpp already uses Thrift 0.12 in some of its binary > > > distributions. So when there is a problem with old readers, one has to > > > notice that we already have files out in the wild. > > > > > > Cheers > > > Uwe > > > > > > On Fri, Jan 25, 2019, at 9:13 AM, Gabor Szadovszky wrote: > > > > May it cause any problems that we write the thrift structures in the > > > > parquet files (footer, page headers etc.) with a different version as > > > > before? It might require some tests if the older readers are able to > > > > read > > > > the files written with the new thrift. > > > > Any thoughts? > > > > > > > > On Thu, Jan 24, 2019 at 8:49 PM Ryan Blue <[email protected]> > > > wrote: > > > > > > > > > Why is it a problem that thrift can't be compiled with Java 11? We > > > should > > > > > only have a binary dependency. > > > > > > > > > > +1 for moving thrift forward, though. > > > > > > > > > > On Thu, Jan 24, 2019 at 11:38 AM Driesprong, Fokko > > > <[email protected]> > > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I would like to discuss updating the Thrift dependency to 0.12.0 of > > > > > > Parquet. In my effort to make Parquet forward compatible for JDK11 > > > > > > <https://github.com/apache/parquet-mr/pull/596>, I stumbled upon > > > some > > > > > > issues. One of them that we still rely, in both the CI and > > > documentation, > > > > > > on Thrift 0.9.3 (released October 2015). Unfortunately, this version > > > of > > > > > > Thrift won't compile under Java11: > > > > > > [javac] Compiling 105 source files to > > > > > /home/travis/build/apache/parquet-mr > > > > > > /thrift-0.9.3/lib/java/build > > > > > > [javac] warning: [options] bootstrap class path not set in > > > conjunction > > > > > with > > > > > > -source 5 > > > > > > [javac] error: Source option 5 is no longer supported. Use 6 or > > > later. > > > > > > [javac] error: Target option 1.5 is no longer supported. Use 1.6 or > > > > > later. > > > > > > > > > > > > Target 1.5, feels a bit awkward, doesn't it? My main question to the > > > > > > dev-list is; is there any particular reason why we shouldn't update > > > the > > > > > > Thrift dependency to 0.12.0. I know that it will have an impact on > > > > > Parquet, > > > > > > but if we want to support Java11, we need to move forward > > > > > > eventually. > > > > > > > > > > > > After updating the thrift-maven plugin > > > > > > <https://github.com/apache/parquet-mr/pull/600>, I was able to run > > > the > > > > > CI > > > > > > against Thrift 0.12.0 <https://github.com/apache/parquet-mr/pull/601 > > > >. > > > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > > > > > > > > > > -- > > > > > Ryan Blue > > > > > Software Engineer > > > > > Netflix > > > > > > > > > > > > > > -- > > Ryan Blue > > Software Engineer > > Netflix
