My concern was related to the current stage of the Variant specification and the fact that we started talking about releasing parquet-java with Variant features. If we formally release parquet-format with the finalized Variant spec first, then I have no concerns about writing Variant values in the upcoming parquet-java release. Otherwise, we need to block it by default and mark it as an experimental feature.
Cheers, Gabor Aihua Xu <aihu...@gmail.com> ezt írta (időpont: 2025. júl. 16., Sze, 19:37): > Hi Gabor and all, > > Here’s my current understanding of the progress on the *Variant* support in > Parquet: > > - > > Per Parquet's requirements, we need at least two reference > implementations to finalize the Variant logical type specification. > - > > The community is actively working on Java, Go, and Rust implementations: > - > > Java already has the encoding and shredding implementations in place: > - > > Variant Decoding < > https://github.com/apache/parquet-java/pull/3197> > - > > Variant Encoding < > https://github.com/apache/parquet-java/pull/3202> > - > > Variant Shredding Writer > <https://github.com/apache/parquet-java/issues/3223> > - > > Variant Shredding Reader > <https://github.com/apache/parquet-java/issues/3211> > - > > Go also includes encoding and shredding support: > - > > Variant Encoding/Decoding > <https://github.com/apache/arrow-go/pull/344> > - > > Variant Shredding <https://github.com/apache/arrow-go/pull/434> > - > > Rust is currently working on the shredding implementation. > > In addition to these, we already have a full Variant implementation in > Apache Iceberg, as well as in some closed-source engines. > > At this point, I’d like to check if we have enough implementation coverage > to move forward with finalizing the Variant spec. Would it make sense to > start a vote thread at this stage? > > Ultimately, our goal is to release a new version of parquet-format and > parquet-java that includes the Variant logical type, so that Iceberg and > other engines can officially depend on it and proceed with further > implementation. > > Let me know your thoughts and how we should proceed. > > Thanks, > > Aihua > > On Sun, Jul 13, 2025 at 10:08 PM Gábor Szádovszky <ga...@apache.org> > wrote: > > > Hi, > > > > I was not able to open the recordings of the last meeting because of > > permission issues. (Shouldn't these be accessible for anyone?) > > So, I'm not sure if you have talked about this, but the Variant spec is > > still not final. Since parquet-java already has Variant support, how do > we > > prevent writing potentially invalid Variant data with the proper logical > > types we will use for the finalized spec? Is it behind a feature flag? > > > > Cheers, > > Gabor > > > > Aihua Xu <aihu...@gmail.com> ezt írta (időpont: 2025. júl. 11., P, > 19:33): > > > > > Hi community, > > > > > > As discussed in the last community sync-up meeting, I'd like to proceed > > > with releasing *Parquet-Java 1.16.0*, which will include support for > > > *geo-type* and *variant*. > > > > > > Please let me know if you have any objections or if you have any > upcoming > > > changes you'd like to include in this release. > > > Thanks, > > > Aihua > > > > > >
