> > Micah, can we perhaps have a separate ML thread for the SemVer proposal?
Done at: https://lists.apache.org/thread/qc031s9vp07bpkrvlnz3v3gn0qhn5l91 On Thu, Jun 25, 2026 at 6:15 AM Antoine Pitrou <[email protected]> wrote: > > > Micah, can we perhaps have a separate ML thread for the SemVer proposal? > > (FTR, I might disagree with how you define SemVer for Parquet :-)) > > Regards > > Antoine. > > > > Le 22/06/2026 à 09:04, Micah Kornfield a écrit : > > To try to move the conversation forward I made two PRs: > > > > 1. parquet-format (https://github.com/apache/parquet-format/pull/588): > > TL;DR; > > - Changes language to use recommended specification version as > the > > mechanism for configuration. > > - Commits to using SemVer for parquet-format releases going > > forward (all forwards incompatible changes, including encodings, etc) > will > > bump the version number. > > - Adds a proposal for a new PARX magic number that has a new > fixed > > length component to the file footer composed of (metadata_len, feature > > bitmap, CRC for footer and 'PARX' trailer). This also unifies encrypted > and > > unencrypted parquet files. > > > > > > 2. A POC in Rust on how this could be implemented ( > > https://github.com/apache/arrow-rs/pull/10177), including its usage with > > path_in_footer > > > > I think this meets the implicit requirements discussed in this thread. > > Namely: > > > > 1. It allows users to think about versions in a canonical way for > feature > > enablement. > > 2. Keep recommendations that won't push default versions too quickly. > > 3. Allows for continuous and iterative releases of the specification. > > 4. Allows flexibility for readers to determine at a granular level if > they > > can properly read the file (and forces a single forward incompatible > change > > so we aren't relying on guesswork for when a writer feature can be safely > > enabled). > > 5. Limits the need for new magic numbers past a single new value. > > > > Please let me know if I missed something. If we can gain consensus > around > > this, I can add a Java implementation so we can adopt the changes and > vote > > on them. > > > > Cheers, > > Micah > > > > > > On Mon, Jun 15, 2026 at 7:45 AM Russell Spitzer < > [email protected]> > > wrote: > > > >> I agree this is getting a bit too complicated, I feel like everyone here > >> understands versions as does the wider community. Why not just start > there > >> and add other techniques if that fails to work properly or be effective > for > >> communication. > >> > >> > >> I think we are better of just choosing something simple and going > forward > >> rather than deliberating, the worst thing that happens is that we have > to > >> make a different choice later. I’m not sure that’s worse than sitting on > >> what we currently have and not being able to make progress on new > encodings > >> or footers. > >> > >> > >> On Mon, Jun 15, 2026 at 3:28 AM Micah Kornfield <[email protected]> > >> wrote: > >> > >>> Thank you for the feedback Andrew. Practically speaking, I wonder if we > >>> should have two separate notions of feature bundling: > >>> > >>> 1. Specification version (this would be primary and risks using > features > >>> that aren't widely adopted). > >>> 2. Presets - Gives users a different way of configuring things that > >> allows > >>> for better guarantees about compatibility in the ecosystem. > >>> > >>> > >>> > >>> On Friday, June 12, 2026, Andrew Bell <[email protected]> > wrote: > >>> > >>>> Hi, > >>>> > >>>> This discussion and the proposals seem to have gotten very > complicated. > >>>> People not on this mailing list or not doing regular development on > >>> Parquet > >>>> would probably benefit from simplicity. Most people are used to > version > >>>> numbers without worrying about various types of compatibility -- > >> readers > >>>> can simply state "I can read version 3-7", for example. Users > >> understand > >>>> this. People writing files can also easily understand "I want to write > >> a > >>>> version 6 file because version 6 supports feature X that I want." or > >> "I > >>>> want to write version 7 because it's the latest version." > >>>> > >>>> I don't really care about the details of a solution, but please keep > in > >>>> mind that a more simple solution probably increases accessibility for > >> the > >>>> widest range of people. > >>>> > >>>> -- > >>>> Andrew Bell > >>>> [email protected] > >>>> > >>> > >> > > > > >
