This is about the binary de/encoding, I think the variant is a separate discussion.
There is a Python implementation here <https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py>, and there seems to be interest from the Arrow/Rust <https://github.com/apache/arrow-rs/issues/6736> community as well. Kind regards, Fokko Op ma 27 jan 2025 om 18:32 schreef Russell Spitzer < russell.spit...@gmail.com>: > The issue we brought up in the discussion was that it's independent of the > Shredding spec which we are also trying to get in. The > python implementation is only for the serialization of a non-shredded > Variant correct? I > think we have our bases covered though with some other language > implementations in the works though > so I'm not sure it's a problem. > > On Mon, Jan 27, 2025 at 11:29 AM Daniel Weeks <dwe...@apache.org> wrote: > > > Yeah, sorry this thread got separated from the original email (for me). > > > > The python implementation currently in Spark is the one I was referring > > to. I believe there is openness to moving that to a separate repository > if > > necessary, but I would also think that if the goal is to prove that there > > is no language specific requirement in the spec, the location of the > > implementation should not matter (e.g. if a rust implementation in Apache > > Arrow would satisfy the requirement, why wouldn't a python implementation > > in Apache Spark)? > > > > -Dan > > > > > > On Sun, Jan 26, 2025 at 4:49 PM Julien Le Dem <jul...@apache.org> wrote: > > > > > Dan, > > > This came from you. > > > Is it the one? > > > Thank you > > > Julien > > > > > > On Sat, Jan 25, 2025 at 11:20 AM Andrew Lamb <andrewlam...@gmail.com> > > > wrote: > > > > > >> Perhaps is the one[1] referred to by Curt[2] > > >> > > >> [1]: > > >> > > >> > > > https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py > > >> [2]: > > >> > https://github.com/apache/arrow-rs/issues/6736#issuecomment-2614029393 > > >> > > >> > > >> > > >> On Wed, Jan 22, 2025 at 12:12 PM Micah Kornfield < > emkornfi...@gmail.com > > > > > >> wrote: > > >> > > >> > What python implementation are we talking about here? > > >> > > > >> > On Wednesday, January 22, 2025, Andrew Lamb <andrewlam...@gmail.com > > > > >> > wrote: > > >> > > > >> > > I also unfortunately will not be able to make it today. > > >> > > > > >> > > As a brief update, I am working on finding someone to help with > the > > >> Rust > > >> > > implementation of variant. Moving forward with Java and Python > seems > > >> > > reasonable to me, though I would truly love to get a Rust > > >> implementation > > >> > to > > >> > > ensure there is no potential gotcha's for a native implementation > > >> > > > > >> > > Thanks, > > >> > > Andrew > > >> > > > > >> > > On Wed, Jan 22, 2025 at 11:41 AM Daniel Weeks <dwe...@apache.org> > > >> wrote: > > >> > > > > >> > > > Hey Julien, > > >> > > > > > >> > > > I'm not going to be able to attend today's meeting, but just > > wanted > > >> to > > >> > > > follow up on a few of the items from the last meeting. > > >> > > > > > >> > > > With respect to the reference implementations for Variant, we > had > > >> > > > discussed the possibility of Rust or C++, but those both have > > >> > significant > > >> > > > work. The Java and native Python implementations are much > closer > > >> and > > >> > > > should cover the concerns for verification of the spec. I still > > >> think > > >> > > > there will be work on the Rust side, but I don't think there's a > > C++ > > >> > > > implementation that would be in a state to open source. > > >> > > > > > >> > > > For the shredding spec, Micah, Ryan, Russel and I met and are > > >> closing > > >> > in > > >> > > on > > >> > > > wording that everyone is happy with, so I expect that will close > > out > > >> > > > shortly. > > >> > > > > > >> > > > -Dan > > >> > > > > > >> > > > On Wed, Jan 22, 2025 at 7:41 AM Julien Le Dem < > jul...@apache.org> > > >> > wrote: > > >> > > > > > >> > > > > The next Parquet sync is today Jan 22nd at 9:30am PT - 12:30pm > > ET > > >> - > > >> > > > 6:30pm > > >> > > > > CET > > >> > > > > (in about 2hs) > > >> > > > > To join the invite: > > >> > > > > https://calendar.app.google/xXGgYU6evBArpzdZ9 > > >> > > > > Please contact me to be added to the recurring invite. > > >> > > > > Everybody is welcome, bring your topic or just listen in. > > >> > > > > Best > > >> > > > > Julien > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
