+0 on accepting Variant into the Parquet *project*, but that's not an approval for sharing repos with the current Parquet format and implementations.
Also, I have the same impression of this vote being a bit prematurate. Is the Variant type as proposed performant enough? Is it flexible enough to enable interoperability with other systems than Spark? Regards Antoine. On Thu, 12 Sep 2024 12:15:13 +0800 Gang Wu <ust...@gmail.com> wrote: > Sorry for the confusion. The intention of this vote is to formally accept > the adoption from Spark and is a formal answer to the corresponding vote > on the Spark side: > https://lists.apache.org/thread/gqy02x1r5dj73woj4l8r0xxkrztd5qos > Both parties should officially agree on the move before discussing the > details. > > On Thu, Sep 12, 2024 at 7:21 AM Jacques Nadeau > <jacques-1odqgaof3lkdnm+yrof...@public.gmane.org> wrote: > > > > My personal opinion is that we've been jumping the gun on voting a bit. I > > > > > do appreciate the enthusiasm though. :) > > > > Yeah, felt the same to me (non-binding). It's hard to fully understand the > > specific actions that are outcomes of this vote. > > > > On Wed, Sep 11, 2024 at 1:11 PM Julien Le Dem > > <julien-1odqgaof3lkdnm+yrof...@public.gmane.org> wrote: > > > > > +1 (binding) on agreeing on principle to add Variant to Parquet. > > > Now on the specific plan, > > > > > > > For repositories to host the Variant specification and library: > > > > - apache/parquet-format will add documentation for the specification > > > > - apache/parquet-java will add a new module for the Java implementation > > > > > > > > > > Gene has posted his doc with a plan "[DISCUSS] Moving Variant to Parquet > > > Details" to collect feedback. > > > Once he's done integrating the feedback and it's finalized, that will be > > > the plan on how to do it. > > > The doc itself is a better reference on how it's going to happen. Maybe > > > that's a better artifact to vote on. > > > > > > My personal opinion is that we've been jumping the gun on voting a bit. I > > > do appreciate the enthusiasm though. :) > > > Voting is more of a procedural mechanism to formally record that we've > > > achieved consensus. > > > > > > > > > > > > > > > > > > On Wed, Sep 11, 2024 at 11:45 AM Micah Kornfield <emkornfi...@gmail.com> > > > wrote: > > > > > > > +1 (binding) in principle on adding it. I think there are still a > > number > > > > of issues to be worked out and we should try to come to a consensus in > > > > Gene's doc [1] + discussion thread on the nitty gritty of what this > > > > proposal actually means. > > > > > > > > [1] > > > > > > > > > > > > > https://docs.google.com/document/d/1guEzBQjzOEEZvvibeZjNraKmZHWtxQR95O_DvtZU0xw/edit#heading=h.5ad5xy8ox6bp > > > > > > > > > > On Wed, Sep 11, 2024 at 9:21 AM Nong Li > > > > <nongli-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote: > > > > > > > > > +1 > > > > > > > > > > On Wed, Sep 11, 2024 at 8:53 AM Gang Wu > > > > > <ustcwg-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote: > > > > > > > > > > > Let's just vote for the adoption in this thread and discuss the > > > > location > > > > > in > > > > > > https://lists.apache.org/thread/xwd3mqjr9bdpg3jcnlprbyb4x09c9ymj > > > > > > > > > > > > Cast my own vote: +1 for adding the variant spec to parquet > > > > > > > > > > > > Thanks! > > > > > > > > > > > > Gang > > > > > > > > > > > > On Wed, Sep 11, 2024 at 5:27 PM Eduard Tudenhöfner < > > > > > > etudenhoef...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > +1 (non-binding) for adding the variant spec to parquet > > > > > > > > > > > > > > On Tue, Sep 10, 2024 at 11:08 PM Daniel Weeks <dwe...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 on adding the variant spec to Parquet > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2024 at 12:23 PM Russell Spitzer < > > > > > > > > russell.spit...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 (Non-binding) This will be great for universal adoption of > > > > > > > > > > > > the > > > > > > > variant > > > > > > > > > type > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2024 at 2:14 PM > > > > > > > > > rdblue-re5jqeeqqe8avxtiumw...@public.gmane.org < > > > > rdb...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 for adding the variant spec to Parquet. I'm looking > > > forward > > > > to > > > > > > > > working > > > > > > > > > > on the addition of shredding. > > > > > > > > > > > > > > > > > > > > As for the details, I think I also prefer a separate > > > > repository, > > > > > > > > > > `parquet-variant`, but I don't think we necessarily need to > > > > > > > > > > > > > > > > determine > > > > > > > > > that > > > > > > > > > > question up front. > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2024 at 9:05 AM Gang Wu <ust...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Antoine, > > > > > > > > > > > > > > > > > > > > > > A separate project was my 1st proposal in the original > > > > > discussion > > > > > > > > > > > on the dev@iceberg ML :). > > > > > > > > > > > > > > > > > > > > > > TBH, I'm open to putting them either in existing repos > > or a > > > > > > > dedicated > > > > > > > > > > > parquet-variant repo. The intention of this thread is to > > > try > > > > to > > > > > > > push > > > > > > > > > the > > > > > > > > > > > discussion to reach a consensus. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2024 at 11:54 PM Antoine Pitrou < > > > > > > > anto...@python.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Gang, > > > > > > > > > > > > > > > > > > > > > > > > Assuming we do want to adopt this in Parquet, I would > > > very > > > > > much > > > > > > > > > > > > recommend separate repositories for this. Putting the > > > spec > > > > > > inside > > > > > > > > > > > > `parquet-format` breeds confusion, IMHO, and may > > > discourage > > > > > > third > > > > > > > > > > > > parties from considering this standalone, non-Parquet, > > > data > > > > > > > format. > > > > > > > > > > > > > > > > > > > > > > > > (but for the same reason, I would recommand a separate > > > > > project > > > > > > as > > > > > > > > > well > > > > > > > > > > > > :-)) > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, 10 Sep 2024 09:48:03 +0800 > > > > > > > > > > > > Gang Wu > > > > > > > > > > > > <ustcwg-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > > > > > I’d like to start a vote for adopting the Variant > > > > > > specification > > > > > > > > and > > > > > > > > > > > > library > > > > > > > > > > > > > from > > > > > > > > > > > > > the Spark project. This allows the Variant binary > > > format > > > > > and > > > > > > > > > > shredding > > > > > > > > > > > > > format > > > > > > > > > > > > > to be more broadly used by other interested projects > > > and > > > > > > > systems. > > > > > > > > > > > > > > > > > > > > > > > > > > For repositories to host the Variant specification > > and > > > > > > library: > > > > > > > > > > > > > - apache/parquet-format will add documentation for > > the > > > > > > > > > specification > > > > > > > > > > > > > - apache/parquet-java will add a new module for the > > > Java > > > > > > > > > > implementation > > > > > > > > > > > > > > > > > > > > > > > > > > Please refer to the discussion thread: > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread/6h58hj39lhqtcyd2hlsyvqm4lzdh4b9z > > > > > > > > > > > > > > > > > > > > > > > > > > This vote will be open for at least 72 hours. > > > > > > > > > > > > > > > > > > > > > > > > > > [ ] +1: Accept the proposal > > > > > > > > > > > > > [ ] +0 > > > > > > > > > > > > > [ ] -1: I don’t think this is a good idea because … > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > Gang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
