Yeah, sorry this thread got separated from the original email (for me). The python implementation currently in Spark is the one I was referring to. I believe there is openness to moving that to a separate repository if necessary, but I would also think that if the goal is to prove that there is no language specific requirement in the spec, the location of the implementation should not matter (e.g. if a rust implementation in Apache Arrow would satisfy the requirement, why wouldn't a python implementation in Apache Spark)?
-Dan On Sun, Jan 26, 2025 at 4:49 PM Julien Le Dem <jul...@apache.org> wrote: > Dan, > This came from you. > Is it the one? > Thank you > Julien > > On Sat, Jan 25, 2025 at 11:20 AM Andrew Lamb <andrewlam...@gmail.com> > wrote: > >> Perhaps is the one[1] referred to by Curt[2] >> >> [1]: >> >> https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py >> [2]: >> https://github.com/apache/arrow-rs/issues/6736#issuecomment-2614029393 >> >> >> >> On Wed, Jan 22, 2025 at 12:12 PM Micah Kornfield <emkornfi...@gmail.com> >> wrote: >> >> > What python implementation are we talking about here? >> > >> > On Wednesday, January 22, 2025, Andrew Lamb <andrewlam...@gmail.com> >> > wrote: >> > >> > > I also unfortunately will not be able to make it today. >> > > >> > > As a brief update, I am working on finding someone to help with the >> Rust >> > > implementation of variant. Moving forward with Java and Python seems >> > > reasonable to me, though I would truly love to get a Rust >> implementation >> > to >> > > ensure there is no potential gotcha's for a native implementation >> > > >> > > Thanks, >> > > Andrew >> > > >> > > On Wed, Jan 22, 2025 at 11:41 AM Daniel Weeks <dwe...@apache.org> >> wrote: >> > > >> > > > Hey Julien, >> > > > >> > > > I'm not going to be able to attend today's meeting, but just wanted >> to >> > > > follow up on a few of the items from the last meeting. >> > > > >> > > > With respect to the reference implementations for Variant, we had >> > > > discussed the possibility of Rust or C++, but those both have >> > significant >> > > > work. The Java and native Python implementations are much closer >> and >> > > > should cover the concerns for verification of the spec. I still >> think >> > > > there will be work on the Rust side, but I don't think there's a C++ >> > > > implementation that would be in a state to open source. >> > > > >> > > > For the shredding spec, Micah, Ryan, Russel and I met and are >> closing >> > in >> > > on >> > > > wording that everyone is happy with, so I expect that will close out >> > > > shortly. >> > > > >> > > > -Dan >> > > > >> > > > On Wed, Jan 22, 2025 at 7:41 AM Julien Le Dem <jul...@apache.org> >> > wrote: >> > > > >> > > > > The next Parquet sync is today Jan 22nd at 9:30am PT - 12:30pm ET >> - >> > > > 6:30pm >> > > > > CET >> > > > > (in about 2hs) >> > > > > To join the invite: >> > > > > https://calendar.app.google/xXGgYU6evBArpzdZ9 >> > > > > Please contact me to be added to the recurring invite. >> > > > > Everybody is welcome, bring your topic or just listen in. >> > > > > Best >> > > > > Julien >> > > > > >> > > > >> > > >> > >> >
