Hi folks, Thanks for those joining the Variant sync yesterday to discuss this proposal. And thanks Kurtis for the note. Appreciate the feedback and discussion as well.
Based on the feedback and the discussion during the call [Notes and Recording here <https://docs.google.com/document/d/1IuhLRxw1rcPD_f4jgHuGe3SwFgy7Y5wgEGvLzf6311s/edit?usp=sharing>]; the proposal should head towards aligning on the best inference algorithm to yield the best results for shredding. I'll be adjusting the proposal as I, along with folks in the community, work on getting the data + testing for the algorithm. Will surface the proposal again with this new direction when it's in a better shape. Thank you and feel free to reach out if you have any questions. On Thu, Jun 4, 2026 at 5:43 PM <[email protected]> wrote: > Hi Neelesh, > > Thank you for organizing the meeting earlier today and starting this > thread. I think it’s awesome to see the community align not just on types > but implementations for Variant. > > I wanted to highlight one additional item I heard discussed on the sync. > > - We need to align on an inference algorithm that defines how a > variant object is shredded. > > > - The goal is we can align using data from high quality benchmarking > data sets if anyone or company is willing to contribute. > > Best Regards, > Kurtis C. Wright > > On Jun 4, 2026, at 09:49, Neelesh Salian <[email protected]> wrote: > > > Hi all, > > Variant shredding writers now exist across iceberg-java, arrow-go, and > arrow-rs, and shredded write paths are landing soon in iceberg-go, > iceberg-rust, and (future) iceberg-python. We don't yet have a shared > policy on whether to shred, how the layout is declared, and how it's > expressed so every implementation produces compatible output. > > I'd like to propose one: Variant Shredding Policy Across Iceberg > Implementations > <https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing> > . > Please take a look. > > Next steps: > 1. The community reviews the proposal and agrees on the shredding > representation. > 2. Once consensus is reached, I'll start a [VOTE] thread. > 3. After the vote passes, I'll land the adjustments in the Java > implementation. > 4. iceberg-go, iceberg-rust, and others follow to align. > > Thanks, > Neelesh > >
