The issue we brought up in the discussion was that it's independent of the Shredding spec which we are also trying to get in. The python implementation is only for the serialization of a non-shredded Variant correct? I think we have our bases covered though with some other language implementations in the works though so I'm not sure it's a problem.
On Mon, Jan 27, 2025 at 11:29 AM Daniel Weeks <dwe...@apache.org> wrote: > Yeah, sorry this thread got separated from the original email (for me). > > The python implementation currently in Spark is the one I was referring > to. I believe there is openness to moving that to a separate repository if > necessary, but I would also think that if the goal is to prove that there > is no language specific requirement in the spec, the location of the > implementation should not matter (e.g. if a rust implementation in Apache > Arrow would satisfy the requirement, why wouldn't a python implementation > in Apache Spark)? > > -Dan > > > On Sun, Jan 26, 2025 at 4:49 PM Julien Le Dem <jul...@apache.org> wrote: > > > Dan, > > This came from you. > > Is it the one? > > Thank you > > Julien > > > > On Sat, Jan 25, 2025 at 11:20 AM Andrew Lamb <andrewlam...@gmail.com> > > wrote: > > > >> Perhaps is the one[1] referred to by Curt[2] > >> > >> [1]: > >> > >> > https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py > >> [2]: > >> https://github.com/apache/arrow-rs/issues/6736#issuecomment-2614029393 > >> > >> > >> > >> On Wed, Jan 22, 2025 at 12:12 PM Micah Kornfield <emkornfi...@gmail.com > > > >> wrote: > >> > >> > What python implementation are we talking about here? > >> > > >> > On Wednesday, January 22, 2025, Andrew Lamb <andrewlam...@gmail.com> > >> > wrote: > >> > > >> > > I also unfortunately will not be able to make it today. > >> > > > >> > > As a brief update, I am working on finding someone to help with the > >> Rust > >> > > implementation of variant. Moving forward with Java and Python seems > >> > > reasonable to me, though I would truly love to get a Rust > >> implementation > >> > to > >> > > ensure there is no potential gotcha's for a native implementation > >> > > > >> > > Thanks, > >> > > Andrew > >> > > > >> > > On Wed, Jan 22, 2025 at 11:41 AM Daniel Weeks <dwe...@apache.org> > >> wrote: > >> > > > >> > > > Hey Julien, > >> > > > > >> > > > I'm not going to be able to attend today's meeting, but just > wanted > >> to > >> > > > follow up on a few of the items from the last meeting. > >> > > > > >> > > > With respect to the reference implementations for Variant, we had > >> > > > discussed the possibility of Rust or C++, but those both have > >> > significant > >> > > > work. The Java and native Python implementations are much closer > >> and > >> > > > should cover the concerns for verification of the spec. I still > >> think > >> > > > there will be work on the Rust side, but I don't think there's a > C++ > >> > > > implementation that would be in a state to open source. > >> > > > > >> > > > For the shredding spec, Micah, Ryan, Russel and I met and are > >> closing > >> > in > >> > > on > >> > > > wording that everyone is happy with, so I expect that will close > out > >> > > > shortly. > >> > > > > >> > > > -Dan > >> > > > > >> > > > On Wed, Jan 22, 2025 at 7:41 AM Julien Le Dem <jul...@apache.org> > >> > wrote: > >> > > > > >> > > > > The next Parquet sync is today Jan 22nd at 9:30am PT - 12:30pm > ET > >> - > >> > > > 6:30pm > >> > > > > CET > >> > > > > (in about 2hs) > >> > > > > To join the invite: > >> > > > > https://calendar.app.google/xXGgYU6evBArpzdZ9 > >> > > > > Please contact me to be added to the recurring invite. > >> > > > > Everybody is welcome, bring your topic or just listen in. > >> > > > > Best > >> > > > > Julien > >> > > > > > >> > > > > >> > > > >> > > >> > > >
