I thought we had agreed it would be merged, and features like this would be feature flag gated. So a V-Old writer could always opt in to using this encoding but that must be a conscious effort if you are not using VNext of Parquet. If you are on V-Next the writer can choose to use ALP whenever it likes without user intervention.
On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <[email protected]> wrote: > Yes, that’s my understanding as well. However, as you noted, the versioning > discussion is still in early stages and hasn’t prescribed concrete > mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no > indication that it’s in preview or that writers shouldn’t use it by default > yet. Given that we’ve held off on merging other incompatible changes (e.g > removal of path_in_schema) until the versioning discussion closes, my > preference would be to leave ALP un-merged for the time being as well. > > — Div > > On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <[email protected]> wrote: > > > After re-reading Julien's Versioning Proposal, my understanding is that > if > > we adopt that system, we will somehow mark ALP in "Preview" in > > parquet-format for release Version 3. > > > > The versioning document doesn't seem to proscribe the particular method > > for annotating "preview" features -- I think clearly marking them as such > > in parquet.thrift as well as reflecting that on the > > https://parquet.apache.org/docs/file-format/versions/ page somehow would > > be adequate. > > > > Andrew > > > > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <[email protected]> > > wrote: > > > >> In my mind ALP is in the same position as other encodings that were > added > >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types) > >> > >> That is to say it will be available in the latest open source writers > and > >> readers, but realistically not widely used in the ecosystem where > >> compatibility is a concern (a large part) > >> > >> I view Julian's work[2][3] to gather consensus on how to name > >> combinations of features (e.g. "versions") as the mechanism by which we > >> will accelerate the adoption of this change (along with other forward > >> incompatible changes) across the ecosystem. > >> > >> > Will we exclude this change when cutting parquet-format v2.14.0? > >> > >> No I don't think so -- I would expect it to go out in the next > >> parquet-format version as do other changes in the spec. > >> > >> Andrew > >> > >> [1]: https://parquet.apache.org/docs/file-format/versions/ > >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss > >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj > >> > >> > >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev < > >> [email protected]> wrote: > >> > >>> Hi Andrew, > >>> > >>> How does this interact with the ongoing versioning discussion? Given > that > >>> new encodings are not forward compatible, I would assume ALP is > included > >>> in > >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's > >>> merged to the parquet-format master branch now. Will we exclude this > >>> change > >>> when cutting parquet-format v2.14.0? > >>> > >>> -- Div > >>> > >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <[email protected]> > >>> wrote: > >>> > >>> > Hello, > >>> > > >>> > As an update, thanks to the epic work from many contributors, > >>> especially > >>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan > to > >>> > make additional editorial The remaining items I know of are merging > the > >>> > examples from parquet-testing[2], and then of course the various > >>> > implementations > >>> > > >>> > Thanks again everyone. This is a great step forward > >>> > Andrew > >>> > > >>> > [1]: https://github.com/apache/parquet-format/pull/557 > >>> > [2]: https://github.com/apache/parquet-testing/pull/100 > >>> > > >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <[email protected]> > >>> wrote: > >>> > > >>> > > It was also in my spam folder... :( > >>> > > I've seen a bunch of apache list emails go to my spam recently. > >>> > > > >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer < > >>> > > [email protected]> > >>> > > wrote: > >>> > > > >>> > > > Yes, my hope was that my vote would kick it out of spam but we > >>> > definitely > >>> > > > had two other votes that also went to spam for me. > >>> > > > > >>> > > > For anyone looking the thread is here > >>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt > >>> > > > > >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield < > >>> > [email protected]> > >>> > > > wrote: > >>> > > > > >>> > > > > I've seen other's votes not go to spam, so hopefully people > will > >>> see > >>> > it > >>> > > > in > >>> > > > > there inboxes (or see this thread and look in there spam). > >>> > > > > > >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR < > >>> [email protected]> > >>> > > > wrote: > >>> > > > > > >>> > > > > > Okay, > >>> > > > > > > >>> > > > > > I sent out the voting email yesterday and so far I've heard > >>> from 3 > >>> > > > people > >>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed > >>> for > >>> > > it?. > >>> > > > > > > >>> > > > > > What's the protocol here? > >>> > > > > > I let people hunt for the email in their spam folder or send > >>> out > >>> > > > another > >>> > > > > > one? > >>> > > > > > > >>> > > > > > Best > >>> > > > > > Prateek > >>> > > > > > > >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR < > >>> [email protected]> > >>> > > > > wrote: > >>> > > > > > > >>> > > > > > > Hi team, > >>> > > > > > > > >>> > > > > > > Vinoo has added the tests which were requested in the last > >>> round > >>> > of > >>> > > > > > review. > >>> > > > > > > In his own words. > >>> > > > > > > > >>> > > > > > > " > >>> > > > > > > The extreme value tests Micah wanted are done and pushed. I > >>> added > >>> > > > > > coverage > >>> > > > > > > for values that need the full FOR bit width after the frame > >>> of > >>> > > > > reference > >>> > > > > > is > >>> > > > > > > applied: 64-bit (the case where the signed max minus min > >>> > > subtraction > >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for > >>> > floats. > >>> > > > All > >>> > > > > > > lossless with zero exceptions. I also verified them through > >>> Arrow > >>> > > C++ > >>> > > > > ALP > >>> > > > > > > reader (the local build), so the Java-written extreme > columns > >>> > > decode > >>> > > > > > > correctly in C++, bit-exact against the expected values." > >>> > > > > > > > >>> > > > > > > State of the PR (#3397): > >>> > > > > > > > >>> > > > > > > - Merges cleanly with master (I pulled master in and > >>> resolved > >>> > > the > >>> > > > > > > conflicts). > >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are > >>> > > > intentional: > >>> > > > > > > one deferred perf nit, one that stays open until > >>> > parquet-format > >>> > > > > ships > >>> > > > > > ALP, > >>> > > > > > > > >>> > > > > > > " > >>> > > > > > > > >>> > > > > > > With this both the > >>> > > > > > > 1) C++ PR : > >>> https://github.com/apache/arrow/pull/48345/changes > >>> > > > > > > 2) Java PR : > >>> https://github.com/apache/parquet-java/pull/3397 > >>> > > > > > > > >>> > > > > > > Are cross language tested and range of test scenarios. > >>> > > > > > > > >>> > > > > > > Best > >>> > > > > > > Prateek > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR < > >>> [email protected] > >>> > > > >>> > > > > wrote: > >>> > > > > > > > >>> > > > > > >> Thanks Matt, > >>> > > > > > >> > >>> > > > > > >> I'll be on it this week. > >>> > > > > > >> > >>> > > > > > >> Best > >>> > > > > > >> Prateek > >>> > > > > > >> > >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol < > >>> > > [email protected]> > >>> > > > > > >> wrote: > >>> > > > > > >> > >>> > > > > > >>> There is also the Go implementation submitted by Arnav ( > >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is > >>> waiting > >>> > > for > >>> > > > > > >>> updates. > >>> > > > > > >>> > >>> > > > > > >>> Just wanted to make sure it didn't get lost here. > >>> > > > > > >>> > >>> > > > > > >>> --Matt > >>> > > > > > >>> > >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR < > >>> > > [email protected]> > >>> > > > > > >>> wrote: > >>> > > > > > >>> > >>> > > > > > >>> > Hi all, > >>> > > > > > >>> > > >>> > > > > > >>> > I'd like to share a status update on the ALP encoding > >>> effort > >>> > > and > >>> > > > > get > >>> > > > > > a > >>> > > > > > >>> feel > >>> > > > > > >>> > from the community before starting a formal vote. > >>> > > > > > >>> > > >>> > > > > > >>> > We now have the specification plus implementations in > >>> both > >>> > > > > languages: > >>> > > > > > >>> > > >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 > >>> Add > >>> > ALP > >>> > > > > > encoding > >>> > > > > > >>> > specification > >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 > >>> > > [C++][Parquet] > >>> > > > > Add > >>> > > > > > >>> ALPpd > >>> > > > > > >>> > encoding > >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet > >>> Java > >>> > ALP > >>> > > > > > >>> > Implementation (by Vinoo) > >>> > > > > > >>> > > >>> > > > > > >>> > Current state: > >>> > > > > > >>> > > >>> > > > > > >>> > - The C++ implementation has been through 3-4 > rounds > >>> of > >>> > > > review. > >>> > > > > > >>> > - The Java implementation has been tracking well > and > >>> has > >>> > > had > >>> > > > > > >>> initial > >>> > > > > > >>> > reviews from contributors, with no major > remaining > >>> > > issues. > >>> > > > > > >>> > - Cross-language compatibility tests are passing: > the > >>> > Arrow > >>> > > > C++ > >>> > > > > > >>> decoder > >>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M > >>> > values > >>> > > > and > >>> > > > > 18 > >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple > vector > >>> > > sizes, > >>> > > > > and > >>> > > > > > >>> > several > >>> > > > > > >>> > real datasets — zero mismatches. > >>> > > > > > >>> > > >>> > > > > > >>> > As raised in the java-pr we will be adding coverage > for > >>> > > extreme > >>> > > > > > values > >>> > > > > > >>> > (those requiring > >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a > vote. > >>> > We'll > >>> > > > aim > >>> > > > > > to > >>> > > > > > >>> get > >>> > > > > > >>> > that > >>> > > > > > >>> > done in parallel. > >>> > > > > > >>> > > >>> > > > > > >>> > Plan: unless there are objections, I intend to start a > >>> vote > >>> > at > >>> > > > the > >>> > > > > > end > >>> > > > > > >>> of > >>> > > > > > >>> > this week/early > >>> > > > > > >>> > next week. > >>> > > > > > >>> > > >>> > > > > > >>> > Thanks > >>> > > > > > >>> > Prateek > >>> > > > > > >>> > > >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR < > >>> > > [email protected]> > >>> > > > > > >>> wrote: > >>> > > > > > >>> > > >>> > > > > > >>> > > Hi Team, > >>> > > > > > >>> > > > >>> > > > > > >>> > > Just wanted to provide some updates on ALP. > >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the > c++ > >>> PR : > >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345 > >>> > > > > > >>> > > > >>> > > > > > >>> > > For the Java implementation Vinoo has been working on > >>> > > following > >>> > > > > PR > >>> > > > > > : > >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397 > >>> > > > > > >>> > > > >>> > > > > > >>> > > Best > >>> > > > > > >>> > > Prateek > >>> > > > > > >>> > > > >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield < > >>> > > > > > >>> [email protected]> > >>> > > > > > >>> > > wrote: > >>> > > > > > >>> > > > >>> > > > > > >>> > >> Hi Antoine, > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> > Apologies if the question was already asked, but > >>> should > >>> > we > >>> > > > > care > >>> > > > > > >>> about > >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient > >>> than > >>> > > > > FLOAT16 > >>> > > > > > + > >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> It was. We thought we could defer it for the > >>> following > >>> > > > reasons: > >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference > >>> > > datasets > >>> > > > to > >>> > > > > > >>> test > >>> > > > > > >>> > its > >>> > > > > > >>> > >> effectiveness. It does look like there might be one > >>> or > >>> > two > >>> > > on > >>> > > > > > >>> > huggingface > >>> > > > > > >>> > >> (e.g. > https://huggingface.co/datasets/kikitora/curdie > >>> ). > >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely > >>> used > >>> > for > >>> > > > > values > >>> > > > > > >>> that > >>> > > > > > >>> > >> were less likely to reduce to decimal values. > >>> > > > > > >>> > >> 3. It could be added as an extension later if > needed. > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> Cheers, > >>> > > > > > >>> > >> Micah > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou < > >>> > > > > [email protected] > >>> > > > > > > > >>> > > > > > >>> > wrote: > >>> > > > > > >>> > >> > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > Hello, > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > Apologies if the question was already asked, but > >>> should > >>> > we > >>> > > > > care > >>> > > > > > >>> about > >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient > >>> than > >>> > > > > FLOAT16 > >>> > > > > > + > >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > Regards > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > Antoine. > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : > >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on > >>> the two > >>> > > > PR's > >>> > > > > > >>> > >> > > 1) (c++ arrow repo) > >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes > >>> > > > > > >>> > >> > > 2) (parquet-format repo) > >>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557 > >>> > > > > > >>> > >> > > > >>> > > > > > >>> > >> > > I have addressed all (unless I missed something) > >>> > > comments > >>> > > > on > >>> > > > > > >>> the two > >>> > > > > > >>> > >> > PR's. > >>> > > > > > >>> > >> > > > >>> > > > > > >>> > >> > > Best > >>> > > > > > >>> > >> > > Prateek > >>> > > > > > >>> > >> > > > >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR < > >>> > > > > > >>> [email protected]> > >>> > > > > > >>> > >> wrote: > >>> > > > > > >>> > >> > > > >>> > > > > > >>> > >> > >> Thanks Andrew and Micah. > >>> > > > > > >>> > >> > >> > >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the > >>> > > implementations` > >>> > > > > > >>> > >> > >> For the c++ I have already started addressing > the > >>> > > > > feedback, I > >>> > > > > > >>> > should > >>> > > > > > >>> > >> be > >>> > > > > > >>> > >> > >> done with that Monday/Tuesday. > >>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress > >>> on > >>> > the > >>> > > > Java > >>> > > > > > >>> > >> > implementation. > >>> > > > > > >>> > >> > >> > >>> > > > > > >>> > >> > >> Best > >>> > > > > > >>> > >> > >> Prateek > >>> > > > > > >>> > >> > >> > >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb < > >>> > > > > > >>> > [email protected] > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > >> wrote: > >>> > > > > > >>> > >> > >> > >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I > >>> will > >>> > try > >>> > > > and > >>> > > > > > look > >>> > > > > > >>> > into > >>> > > > > > >>> > >> the > >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this > next > >>> > week > >>> > > > > > >>> > >> > >>> > >>> > > > > > >>> > >> > >>> [1]: > >>> https://github.com/apache/arrow-rs/pull/9372 > >>> > > > > > >>> > >> > >>> > >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah > >>> Kornfield < > >>> > > > > > >>> > >> > [email protected]> > >>> > > > > > >>> > >> > >>> wrote: > >>> > > > > > >>> > >> > >>> > >>> > > > > > >>> > >> > >>>> Hi Andrew, > >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on > >>> at > >>> > > least > >>> > > > > the > >>> > > > > > >>> > >> > >>>> implementations, typically I think we've > waited > >>> > till > >>> > > > they > >>> > > > > > are > >>> > > > > > >>> > >> close to > >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I > >>> agree > >>> > we > >>> > > > are > >>> > > > > > very > >>> > > > > > >>> > >> close. > >>> > > > > > >>> > >> > >>>> > >>> > > > > > >>> > >> > >>>> -Micah > >>> > > > > > >>> > >> > >>>> > >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb < > >>> > > > > > >>> [email protected] > >>> > > > > > >>> > > > >>> > > > > > >>> > >> > wrote: > >>> > > > > > >>> > >> > >>>> > >>> > > > > > >>> > >> > >>>>> Thanks Prateek, > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me > like > >>> we > >>> > are > >>> > > > > ready > >>> > > > > > >>> to > >>> > > > > > >>> > >> start a > >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should > >>> postpone it > >>> > > for > >>> > > > > > >>> longer? > >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to > >>> review? > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> Andrew > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK > GAUR < > >>> > > > > > >>> > [email protected]> > >>> > > > > > >>> > >> > >>>>> wrote: > >>> > > > > > >>> > >> > >>>>> > >>> > > > > > >>> > >> > >>>>>> Hi team, > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance > >>> to > >>> > work > >>> > > > > > >>> through all > >>> > > > > > >>> > >> the > >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. > >>> Here > >>> > > are > >>> > > > > the > >>> > > > > > >>> new > >>> > > > > > >>> > >> > artifacts > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> 1) Spec document : > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > >>> > > > > > >>> > > >>> > > > > > >>> > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo : > >>> > > > > > >>> > >> > >>>>>> > >>> > https://github.com/apache/parquet-format/pull/557 > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo : > >>> > > > > > >>> > >> > >>>>>> > >>> > https://github.com/apache/arrow/pull/48345/changes > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo > : > >>> Work > >>> > > for > >>> > > > > > >>> Vinoo and > >>> > > > > > >>> > >> > Julien > >>> > > > > > >>> > >> > >>>>>> > >>> > https://github.com/apache/parquet-java/pull/3397 > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts > in > >>> > > > > > >>> parquet-testing > >>> > > > > > >>> > >> repo : > >>> > > > > > >>> > >> > >>>>>> > >>> > https://github.com/apache/parquet-testing/pull/100 > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> And > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in > >>> progress > >>> > > > > > >>> implementation > >>> > > > > > >>> > in > >>> > > > > > >>> > >> Go. > >>> > > > > > >>> > >> > >>>>>> > >>> https://github.com/apache/arrow-go/pull/704 > >>> > (I > >>> > > > > > haven't > >>> > > > > > >>> > >> started > >>> > > > > > >>> > >> > >>>>>> looking at it yet) > >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned > that > >>> > this > >>> > > > work > >>> > > > > > is > >>> > > > > > >>> also > >>> > > > > > >>> > >> in > >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!) > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation * > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine > to > >>> > > report > >>> > > > > the > >>> > > > > > >>> > numbers > >>> > > > > > >>> > >> as > >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have > >>> given 1 > >>> > > > round > >>> > > > > of > >>> > > > > > >>> > >> feedback > >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note > >>> that > >>> > the > >>> > > > > > default > >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and > not > >>> > Q3. I > >>> > > > got > >>> > > > > > >>> around > >>> > > > > > >>> > >> 70% > >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding > speed > >>> > when > >>> > > > > using > >>> > > > > > >>> the O3 > >>> > > > > > >>> > >> > flag. > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working > with > >>> > Vinoo > >>> > > > and > >>> > > > > > >>> Julien) > >>> > > > > > >>> > >> and > >>> > > > > > >>> > >> > **Cross > >>> > > > > > >>> > >> > >>>>>> Language testing* > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions > or > >>> > > > feedback. > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify > >>> > Columns > >>> > > > > > >>> (Graviton > >>> > > > > > >>> > 3, > >>> > > > > > >>> > >> ARM > >>> > > > > > >>> > >> > >>>>>> Neoverse V1) > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 > >>> > > (MB/s) │ > >>> > > > > > >>> Speedup │ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ > >>> 5,523 > >>> > > │ > >>> > > > > > >>> 1.75x │ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ > >>> 5,685 > >>> > > │ > >>> > > > > > >>> 1.76x │ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ > >>> 5,652 > >>> > > │ > >>> > > > > > >>> 1.77x │ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ > >>> 5,473 > >>> > > │ > >>> > > > > > >>> 1.72x │ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >>> └──────────────────┴──────────────┴──────────────┴─────────┘ > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK > GAUR > >>> < > >>> > > > > > >>> > [email protected] > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > >>>>>> wrote: > >>> > > > > > >>> > >> > >>>>>> > >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <[email protected]> > : > >>> Got > >>> > > it. > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <[email protected]> > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start > >>> moving > >>> > the > >>> > > > > spec > >>> > > > > > >>> > >> development > >>> > > > > > >>> > >> > >>>>>>>> into > >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for > >>> finalizing > >>> > > it? > >>> > > > > > >>> > >> > >>>>>>>> > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of > the > >>> > > > examples I > >>> > > > > > >>> have in > >>> > > > > > >>> > >> the > >>> > > > > > >>> > >> > >>>>>>> spec based > >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should > >>> be > >>> > good > >>> > > > to > >>> > > > > go > >>> > > > > > >>> for > >>> > > > > > >>> > the > >>> > > > > > >>> > >> > >>>>>>> markdown format. > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>> Thanks everyone! > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > >>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>> Andrew > >>> > > > > > >>> > >> > >>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK > >>> GAUR < > >>> > > > > > >>> > >> [email protected]> > >>> > > > > > >>> > >> > >>>>>>>> wrote: > >>> > > > > > >>> > >> > >>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> Hi team, > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. > >>> My PR > >>> > > did > >>> > > > > add > >>> > > > > > >>> all > >>> > > > > > >>> > the > >>> > > > > > >>> > >> > test > >>> > > > > > >>> > >> > >>>>>>>> files I > >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe > >>> we > >>> > can > >>> > > > club > >>> > > > > > it > >>> > > > > > >>> > >> together. > >>> > > > > > >>> > >> > >>>>>>>> WIll also > >>> > > > > > >>> > >> > >>>>>>>>> aid > >>> > > > > > >>> > >> > >>>>>>>>> cross language testing > >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust > >>> > > > implementation. > >>> > > > > > >>> This is > >>> > > > > > >>> > >> > great. > >>> > > > > > >>> > >> > >>>>>>>> Thanks > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the > >>> numbers > >>> > on > >>> > > > AMD. > >>> > > > > > >>> Looks > >>> > > > > > >>> > >> like > >>> > > > > > >>> > >> > you > >>> > > > > > >>> > >> > >>>>>>>> are > >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance > of > >>> > BSS. > >>> > > > This > >>> > > > > > is > >>> > > > > > >>> > >> > amazing!!. > >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the > >>> sampling > >>> > > > design. > >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In > >>> some > >>> > > crude > >>> > > > > > >>> > >> experiments I > >>> > > > > > >>> > >> > >>>>>>>> didn't get > >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on > >>> Graviton3 > >>> > > (but > >>> > > > > > maybe > >>> > > > > > >>> > there > >>> > > > > > >>> > >> was > >>> > > > > > >>> > >> > >>>>>>>> something > >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation). > >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception > >>> encoding for > >>> > > the > >>> > > > > > spec > >>> > > > > > >>> in > >>> > > > > > >>> > >> this > >>> > > > > > >>> > >> > >>>>>>>> case. > >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for > >>> all > >>> > > open > >>> > > > > > >>> questions > >>> > > > > > >>> > >> minus > >>> > > > > > >>> > >> > >>>>>>>> the > >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on > >>> this > >>> > > soon) > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does > >>> allow > >>> > > for > >>> > > > > > using > >>> > > > > > >>> > >> FastLane > >>> > > > > > >>> > >> > >>>>>>>> with the > >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. > We > >>> > > should > >>> > > > be > >>> > > > > > >>> able to > >>> > > > > > >>> > >> > inject > >>> > > > > > >>> > >> > >>>>>>>> any > >>> > > > > > >>> > >> > >>>>>>>>> layout > >>> > > > > > >>> > >> > >>>>>>>>> in the current design. > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open > >>> > comments > >>> > > > on > >>> > > > > > the > >>> > > > > > >>> spec > >>> > > > > > >>> > >> this > >>> > > > > > >>> > >> > >>>>>>>> week. > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> Best > >>> > > > > > >>> > >> > >>>>>>>>> Prateek > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve > >>> > Loughran < > >>> > > > > > >>> > >> > >>>>>>>> [email protected]> > >>> > > > > > >>> > >> > >>>>>>>>> wrote: > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah > >>> Kornfield > >>> > < > >>> > > > > > >>> > >> > >>>>>>>> [email protected]> > >>> > > > > > >>> > >> > >>>>>>>>>> wrote: > >>> > > > > > >>> > >> > >>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue > >>> described for > >>> > > ORC > >>> > > > > in > >>> > > > > > >>> the > >>> > > > > > >>> > >> paper > >>> > > > > > >>> > >> > >>>>>>>> is that > >>> > > > > > >>> > >> > >>>>>>>>> it > >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. > >>> This > >>> > > is > >>> > > > > > >>> different > >>> > > > > > >>> > >> then > >>> > > > > > >>> > >> > >>>>>>>> the > >>> > > > > > >>> > >> > >>>>>>>>> design > >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still > fixed > >>> > > > encoding > >>> > > > > > per > >>> > > > > > >>> page > >>> > > > > > >>> > >> in > >>> > > > > > >>> > >> > >>>>>>>> parquet. > >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't > >>> think > >>> > > > branch > >>> > > > > > >>> > >> > >>>>>>>> misprediction should > >>> > > > > > >>> > >> > >>>>>>>>>> be a > >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree > >>> that > >>> > we > >>> > > > > should > >>> > > > > > >>> be > >>> > > > > > >>> > >> > >>>>>>>> conservative in > >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings. > >>> > > > > > >>> > >> > >>>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>>> +1 > >>> > > > > > >>> > >> > >>>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>>> > >>> > > > > > >>> > >> > >>>>>>>> > >>> > > > > > >>> > >> > >>>>>>> > >>> > > > > > >>> > >> > > > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > > >>> > > > > > >>> > >> > >>> > > > > > >>> > > > >>> > > > > > >>> > > >>> > > > > > >>> > >>> > > > > > >> > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> >
