Hi folks, I’ve updated the document to track the Variant work [here <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>]. Please refer to the “Next milestones” section to see what is coming. I’ve cancelled the call for December due to the holidays, we’ll catch up in January 2027.
Regardless of the sync, I’ll continue to keep that document updated as I go across the communities that work on Variant. Please don’t hesitate to reach out to me if you have questions around Variant and how we are progressing. On Sun, Sep 6, 2026 at 10:44 AM Neelesh Salian <[email protected]> wrote: > Hi folks, > > Here is the Meeting Notes and the Recording link from the Variant Sync on > September 03, 2026: [Notes > <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> > ]. > Please reach out if you have any questions. > Thanks. > > On Sun, Aug 9, 2026 at 5:49 PM Neelesh Salian <[email protected]> > wrote: > >> Hi folks, >> >> Here is the Meeting Notes and Recording link from the Variant Sync on >> August 06, 2026: [Notes >> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >> ]. >> Please reach out if you have any questions. >> Thanks. >> >> On Thu, Jul 2, 2026 at 8:21 PM Neelesh Salian <[email protected]> >> wrote: >> >>> Hi folks, >>> >>> Here is the Meeting Notes and Recording link from the Variant Sync today >>> July 02, 2026: [Notes >>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >>> ]. >>> There are some action items and active work. >>> Please reach out if you have any questions. >>> Thanks >>> >>> On Fri, Jun 5, 2026 at 12:22 PM Neelesh Salian <[email protected]> >>> wrote: >>> >>>> Hi folks, >>>> >>>> Here is the Meeting Notes and Recording link from the Variant Sync on >>>> June 4, 2026: [Notes >>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz#heading=h.r977qio1wsv2> >>>> ]. >>>> There are some action items and active work. >>>> Please reach out if you have any questions. >>>> Thanks. >>>> >>>> On Fri, May 8, 2026 at 3:20 PM Neelesh Salian <[email protected]> >>>> wrote: >>>> >>>>> Hi folks, >>>>> >>>>> Here is the Meeting Notes and Recording link from the Variant Sync on >>>>> May 7, 2026: [Notes >>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?tab=t.g06kaw1fbwhz> >>>>> ]. >>>>> There are some action items and active work. >>>>> Please reach out if you have any questions. >>>>> Thanks. >>>>> >>>>> On Thu, Apr 30, 2026 at 1:36 PM Neelesh Salian < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi folks, >>>>>> >>>>>> I've set up a time starting next week on Thursday (May 7, 2026) at 10 >>>>>> am Pacific time for a sync for the active work on Variant. >>>>>> This will be a monthly sync (on the first Thursday of every month). >>>>>> You can find it on the dev calendar. >>>>>> Here is the calendar invite: >>>>>> https://calendar.app.google/b8ykdTV3EaNnVnkv8 >>>>>> I'll be recording the call and capturing notes in the sync document: >>>>>> Iceberg >>>>>> - Variant Community Update >>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> >>>>>> (Meeting >>>>>> Notes tab). >>>>>> Thanks. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Apr 20, 2026 at 1:49 PM Steve Loughran <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> + regarding the rust, go and cpp impls, a status from each team >>>>>>> would be great! >>>>>>> >>>>>>> I've been reviewing arrow parquet variant stuff and it is all there, >>>>>>> including with some benchmarks and optimisations. Which may put it >>>>>>> ahead of >>>>>>> the others. >>>>>>> >>>>>>> It also has some special handling for sorted variants, as key search >>>>>>> there is straightforward. AFAIK I don't think the others do that, and >>>>>>> nor >>>>>>> do I see them going to any effort to sort fields in an object. I think >>>>>>> sorting would be good, but you would have to handle the case where there >>>>>>> are duplicate keys. It's allowed in the spec, and seems like itcould >>>>>>> creep >>>>>>> in from nested variants. Has anyone looked at this? >>>>>>> >>>>>>> Also: has anyone created malformed parquet files with a shredded >>>>>>> variant and a metadata entry of the same name. The requirement is >>>>>>> "ignore >>>>>>> the metadata one", but that's something to test. You'd have to write a >>>>>>> shredded file and then edit the binary content to achieve this, or >>>>>>> manually >>>>>>> create one and put it into the parquet-testing repository under >>>>>>> bad-data/ >>>>>>> >>>>>>> >>>>>>> On Mon, 20 Apr 2026 at 19:08, Qiegang Long <[email protected]> wrote: >>>>>>> >>>>>>>> Thanks for the doc to track the status! +1 on the dedicated >>>>>>>> sync—definitely feels like there’s a lot of work before we see >>>>>>>> Variant’s >>>>>>>> full potential. >>>>>>>> >>>>>>>> Qiegang >>>>>>>> >>>>>>>> On Mon, Apr 20, 2026 at 11:09 AM Steve Loughran < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> This is great, we need that tracker as it is cross-project. piece >>>>>>>>> of work to say "this is readly >>>>>>>>> >>>>>>>>> I did have an agenda item from last month's community call which >>>>>>>>> didn't get through. If we can retain that open time slot we could do >>>>>>>>> a very >>>>>>>>> quick summary of where we are (summarly slides of Qiegang's results >>>>>>>>> and >>>>>>>>> mine, key outstanding issues and next steps, then we can start that >>>>>>>>> monthly session on it. >>>>>>>>> >>>>>>>>> Meanwhile, I have both parquet and iceberg PRs for >>>>>>>>> benchmarks which I think are ready for review -please take a look >>>>>>>>> >>>>>>>>> Finally, I'm thinking about interop of those many, many variant >>>>>>>>> readers out there. Has anyone explicitly cross-tested their >>>>>>>>> implementations >>>>>>>>> of variant? what about consistent handling of invalid data? That >>>>>>>>> includes >>>>>>>>> iceberg-rust, parquet-cpp and more... >>>>>>>>> >>>>>>>>> Steve >>>>>>>>> >>>>>>>>> On Sun, 19 Apr 2026 at 21:57, Neelesh Salian < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi everyone, >>>>>>>>>> >>>>>>>>>> The Variant umbrella issue (#10392 >>>>>>>>>> <https://github.com/apache/iceberg/issues/10392>) hasn't been >>>>>>>>>> updated in a while, and with active work happening across multiple >>>>>>>>>> PRs in >>>>>>>>>> Iceberg, Spark, and Parquet, it's been hard to keep track of where >>>>>>>>>> things >>>>>>>>>> stand. >>>>>>>>>> >>>>>>>>>> Since a few of us are actively working on variant features, I >>>>>>>>>> thought it would help to put together a tracking document so the >>>>>>>>>> community >>>>>>>>>> has a single place to see the current state, open work, and benchmark >>>>>>>>>> findings. I plan to update this on a weekly basis to keep track of >>>>>>>>>> the >>>>>>>>>> issues and PRs that are updated. >>>>>>>>>> >>>>>>>>>> Iceberg Variant Community Document >>>>>>>>>> <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing> >>>>>>>>>> >>>>>>>>>> The document has three tabs: >>>>>>>>>> >>>>>>>>>> 1. Overview - what shipped in 1.10, what's merged to main, >>>>>>>>>> open work areas, and the dependency graph across Iceberg, Spark, >>>>>>>>>> and Parquet >>>>>>>>>> 2. Tracker - all open variant issues and PRs across Iceberg, >>>>>>>>>> Parquet-Java, Parquet-Format, and Spark with authors and status >>>>>>>>>> 3. Benchmarks - summary of three independent benchmark >>>>>>>>>> efforts (details below) >>>>>>>>>> >>>>>>>>>> *Benchmark findings* >>>>>>>>>> >>>>>>>>>> Three independent benchmarks have measured variant performance. >>>>>>>>>> All converge on the same picture: variant is a modest improvement >>>>>>>>>> over JSON >>>>>>>>>> strings today (1.1-1.7x faster reads), but 15-17x slower than typed >>>>>>>>>> columns. >>>>>>>>>> >>>>>>>>>> 1. Qiegang Long - 14 queries on GitHub Archive, 5 configs: >>>>>>>>>> https://qlong.github.io/posts/2026-03-30-variant-early-results >>>>>>>>>> 2. Steve Loughran - JMH microbenchmarks, profiler-driven >>>>>>>>>> optimization: >>>>>>>>>> >>>>>>>>>> https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html >>>>>>>>>> >>>>>>>>>> <https://steveloughran.github.io/benchmarking-variants/benchmarking-variants.html> >>>>>>>>>> 3. Neelesh Salian - Controlled baseline, 10M+100M rows, write >>>>>>>>>> + read: >>>>>>>>>> >>>>>>>>>> https://github.com/nssalian/iceberg/tree/iceberg-variant-benchmark/benchmark >>>>>>>>>> >>>>>>>>>> If you're working on variant-related changes, please chime in or >>>>>>>>>> let me know and I'll add it to the tracker. Feedback on the >>>>>>>>>> benchmarks or >>>>>>>>>> anything else is welcome. >>>>>>>>>> >>>>>>>>>> I've been giving variant updates during the Iceberg Spark Sync >>>>>>>>>> (Tuesdays, 10 AM PT), but given that this work now spans Iceberg, >>>>>>>>>> Spark, >>>>>>>>>> Parquet, and Flink, I think it deserves its own forum. I'd like to >>>>>>>>>> propose >>>>>>>>>> a monthly Variant Sync; a short call where contributors can share >>>>>>>>>> progress, >>>>>>>>>> surface blockers, and coordinate across repos. If there's interest, >>>>>>>>>> I'll >>>>>>>>>> set one up and share an invite on this thread. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Neelesh Salian. >>>>>>>>>> >>>>>>>>>
