Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in the spec section).
Russel: That is my understanding as well. Hopefully we are able to formalize the versioning process shortly so we can formalize the notion of "preview" Andrew [1]: https://github.com/apache/parquet-format/pull/604 On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <[email protected]> wrote: > I thought we had agreed it would be merged, and features like this would > be feature flag gated. So a V-Old writer could always opt in to using this > encoding but that must be a conscious effort if you are not using VNext of > Parquet. If you are on V-Next the writer can choose to use ALP whenever it > likes without user intervention. > > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev < > [email protected]> wrote: > >> Yes, that’s my understanding as well. However, as you noted, the >> versioning >> discussion is still in early stages and hasn’t prescribed concrete >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no >> indication that it’s in preview or that writers shouldn’t use it by >> default >> yet. Given that we’ve held off on merging other incompatible changes (e.g >> removal of path_in_schema) until the versioning discussion closes, my >> preference would be to leave ALP un-merged for the time being as well. >> >> — Div >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <[email protected]> wrote: >> >> > After re-reading Julien's Versioning Proposal, my understanding is that >> if >> > we adopt that system, we will somehow mark ALP in "Preview" in >> > parquet-format for release Version 3. >> > >> > The versioning document doesn't seem to proscribe the particular method >> > for annotating "preview" features -- I think clearly marking them as >> such >> > in parquet.thrift as well as reflecting that on the >> > https://parquet.apache.org/docs/file-format/versions/ page somehow >> would >> > be adequate. >> > >> > Andrew >> > >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <[email protected]> >> > wrote: >> > >> >> In my mind ALP is in the same position as other encodings that were >> added >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types) >> >> >> >> That is to say it will be available in the latest open source writers >> and >> >> readers, but realistically not widely used in the ecosystem where >> >> compatibility is a concern (a large part) >> >> >> >> I view Julian's work[2][3] to gather consensus on how to name >> >> combinations of features (e.g. "versions") as the mechanism by which we >> >> will accelerate the adoption of this change (along with other forward >> >> incompatible changes) across the ecosystem. >> >> >> >> > Will we exclude this change when cutting parquet-format v2.14.0? >> >> >> >> No I don't think so -- I would expect it to go out in the next >> >> parquet-format version as do other changes in the spec. >> >> >> >> Andrew >> >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/ >> >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss >> >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj >> >> >> >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev < >> >> [email protected]> wrote: >> >> >> >>> Hi Andrew, >> >>> >> >>> How does this interact with the ongoing versioning discussion? Given >> that >> >>> new encodings are not forward compatible, I would assume ALP is >> included >> >>> in >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's >> >>> merged to the parquet-format master branch now. Will we exclude this >> >>> change >> >>> when cutting parquet-format v2.14.0? >> >>> >> >>> -- Div >> >>> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <[email protected]> >> >>> wrote: >> >>> >> >>> > Hello, >> >>> > >> >>> > As an update, thanks to the epic work from many contributors, >> >>> especially >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We >> plan to >> >>> > make additional editorial The remaining items I know of are merging >> the >> >>> > examples from parquet-testing[2], and then of course the various >> >>> > implementations >> >>> > >> >>> > Thanks again everyone. This is a great step forward >> >>> > Andrew >> >>> > >> >>> > [1]: https://github.com/apache/parquet-format/pull/557 >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100 >> >>> > >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <[email protected]> >> >>> wrote: >> >>> > >> >>> > > It was also in my spam folder... :( >> >>> > > I've seen a bunch of apache list emails go to my spam recently. >> >>> > > >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer < >> >>> > > [email protected]> >> >>> > > wrote: >> >>> > > >> >>> > > > Yes, my hope was that my vote would kick it out of spam but we >> >>> > definitely >> >>> > > > had two other votes that also went to spam for me. >> >>> > > > >> >>> > > > For anyone looking the thread is here >> >>> > > > >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt >> >>> > > > >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield < >> >>> > [email protected]> >> >>> > > > wrote: >> >>> > > > >> >>> > > > > I've seen other's votes not go to spam, so hopefully people >> will >> >>> see >> >>> > it >> >>> > > > in >> >>> > > > > there inboxes (or see this thread and look in there spam). >> >>> > > > > >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR < >> >>> [email protected]> >> >>> > > > wrote: >> >>> > > > > >> >>> > > > > > Okay, >> >>> > > > > > >> >>> > > > > > I sent out the voting email yesterday and so far I've heard >> >>> from 3 >> >>> > > > people >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be >> blamed >> >>> for >> >>> > > it?. >> >>> > > > > > >> >>> > > > > > What's the protocol here? >> >>> > > > > > I let people hunt for the email in their spam folder or send >> >>> out >> >>> > > > another >> >>> > > > > > one? >> >>> > > > > > >> >>> > > > > > Best >> >>> > > > > > Prateek >> >>> > > > > > >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR < >> >>> [email protected]> >> >>> > > > > wrote: >> >>> > > > > > >> >>> > > > > > > Hi team, >> >>> > > > > > > >> >>> > > > > > > Vinoo has added the tests which were requested in the last >> >>> round >> >>> > of >> >>> > > > > > review. >> >>> > > > > > > In his own words. >> >>> > > > > > > >> >>> > > > > > > " >> >>> > > > > > > The extreme value tests Micah wanted are done and pushed. >> I >> >>> added >> >>> > > > > > coverage >> >>> > > > > > > for values that need the full FOR bit width after the >> frame >> >>> of >> >>> > > > > reference >> >>> > > > > > is >> >>> > > > > > > applied: 64-bit (the case where the signed max minus min >> >>> > > subtraction >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for >> >>> > floats. >> >>> > > > All >> >>> > > > > > > lossless with zero exceptions. I also verified them >> through >> >>> Arrow >> >>> > > C++ >> >>> > > > > ALP >> >>> > > > > > > reader (the local build), so the Java-written extreme >> columns >> >>> > > decode >> >>> > > > > > > correctly in C++, bit-exact against the expected values." >> >>> > > > > > > >> >>> > > > > > > State of the PR (#3397): >> >>> > > > > > > >> >>> > > > > > > - Merges cleanly with master (I pulled master in and >> >>> resolved >> >>> > > the >> >>> > > > > > > conflicts). >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are >> >>> > > > intentional: >> >>> > > > > > > one deferred perf nit, one that stays open until >> >>> > parquet-format >> >>> > > > > ships >> >>> > > > > > ALP, >> >>> > > > > > > >> >>> > > > > > > " >> >>> > > > > > > >> >>> > > > > > > With this both the >> >>> > > > > > > 1) C++ PR : >> >>> https://github.com/apache/arrow/pull/48345/changes >> >>> > > > > > > 2) Java PR : >> >>> https://github.com/apache/parquet-java/pull/3397 >> >>> > > > > > > >> >>> > > > > > > Are cross language tested and range of test scenarios. >> >>> > > > > > > >> >>> > > > > > > Best >> >>> > > > > > > Prateek >> >>> > > > > > > >> >>> > > > > > > >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR < >> >>> [email protected] >> >>> > > >> >>> > > > > wrote: >> >>> > > > > > > >> >>> > > > > > >> Thanks Matt, >> >>> > > > > > >> >> >>> > > > > > >> I'll be on it this week. >> >>> > > > > > >> >> >>> > > > > > >> Best >> >>> > > > > > >> Prateek >> >>> > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol < >> >>> > > [email protected]> >> >>> > > > > > >> wrote: >> >>> > > > > > >> >> >>> > > > > > >>> There is also the Go implementation submitted by Arnav ( >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is >> >>> waiting >> >>> > > for >> >>> > > > > > >>> updates. >> >>> > > > > > >>> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here. >> >>> > > > > > >>> >> >>> > > > > > >>> --Matt >> >>> > > > > > >>> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR < >> >>> > > [email protected]> >> >>> > > > > > >>> wrote: >> >>> > > > > > >>> >> >>> > > > > > >>> > Hi all, >> >>> > > > > > >>> > >> >>> > > > > > >>> > I'd like to share a status update on the ALP encoding >> >>> effort >> >>> > > and >> >>> > > > > get >> >>> > > > > > a >> >>> > > > > > >>> feel >> >>> > > > > > >>> > from the community before starting a formal vote. >> >>> > > > > > >>> > >> >>> > > > > > >>> > We now have the specification plus implementations in >> >>> both >> >>> > > > > languages: >> >>> > > > > > >>> > >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 >> >>> Add >> >>> > ALP >> >>> > > > > > encoding >> >>> > > > > > >>> > specification >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 >> >>> > > [C++][Parquet] >> >>> > > > > Add >> >>> > > > > > >>> ALPpd >> >>> > > > > > >>> > encoding >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet >> >>> Java >> >>> > ALP >> >>> > > > > > >>> > Implementation (by Vinoo) >> >>> > > > > > >>> > >> >>> > > > > > >>> > Current state: >> >>> > > > > > >>> > >> >>> > > > > > >>> > - The C++ implementation has been through 3-4 >> rounds >> >>> of >> >>> > > > review. >> >>> > > > > > >>> > - The Java implementation has been tracking well >> and >> >>> has >> >>> > > had >> >>> > > > > > >>> initial >> >>> > > > > > >>> > reviews from contributors, with no major >> remaining >> >>> > > issues. >> >>> > > > > > >>> > - Cross-language compatibility tests are passing: >> the >> >>> > Arrow >> >>> > > > C++ >> >>> > > > > > >>> decoder >> >>> > > > > > >>> > reads Java-written data bit-exactly across >> ~1.56M >> >>> > values >> >>> > > > and >> >>> > > > > 18 >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple >> vector >> >>> > > sizes, >> >>> > > > > and >> >>> > > > > > >>> > several >> >>> > > > > > >>> > real datasets — zero mismatches. >> >>> > > > > > >>> > >> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage >> for >> >>> > > extreme >> >>> > > > > > values >> >>> > > > > > >>> > (those requiring >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a >> vote. >> >>> > We'll >> >>> > > > aim >> >>> > > > > > to >> >>> > > > > > >>> get >> >>> > > > > > >>> > that >> >>> > > > > > >>> > done in parallel. >> >>> > > > > > >>> > >> >>> > > > > > >>> > Plan: unless there are objections, I intend to start a >> >>> vote >> >>> > at >> >>> > > > the >> >>> > > > > > end >> >>> > > > > > >>> of >> >>> > > > > > >>> > this week/early >> >>> > > > > > >>> > next week. >> >>> > > > > > >>> > >> >>> > > > > > >>> > Thanks >> >>> > > > > > >>> > Prateek >> >>> > > > > > >>> > >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR < >> >>> > > [email protected]> >> >>> > > > > > >>> wrote: >> >>> > > > > > >>> > >> >>> > > > > > >>> > > Hi Team, >> >>> > > > > > >>> > > >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP. >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the >> c++ >> >>> PR : >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345 >> >>> > > > > > >>> > > >> >>> > > > > > >>> > > For the Java implementation Vinoo has been working >> on >> >>> > > following >> >>> > > > > PR >> >>> > > > > > : >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397 >> >>> > > > > > >>> > > >> >>> > > > > > >>> > > Best >> >>> > > > > > >>> > > Prateek >> >>> > > > > > >>> > > >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield < >> >>> > > > > > >>> [email protected]> >> >>> > > > > > >>> > > wrote: >> >>> > > > > > >>> > > >> >>> > > > > > >>> > >> Hi Antoine, >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked, but >> >>> should >> >>> > we >> >>> > > > > care >> >>> > > > > > >>> about >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more >> efficient >> >>> than >> >>> > > > > FLOAT16 >> >>> > > > > > + >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the >> >>> following >> >>> > > > reasons: >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy >> reference >> >>> > > datasets >> >>> > > > to >> >>> > > > > > >>> test >> >>> > > > > > >>> > its >> >>> > > > > > >>> > >> effectiveness. It does look like there might be >> one >> >>> or >> >>> > two >> >>> > > on >> >>> > > > > > >>> > huggingface >> >>> > > > > > >>> > >> (e.g. >> https://huggingface.co/datasets/kikitora/curdie >> >>> ). >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely >> >>> used >> >>> > for >> >>> > > > > values >> >>> > > > > > >>> that >> >>> > > > > > >>> > >> were less likely to reduce to decimal values. >> >>> > > > > > >>> > >> 3. It could be added as an extension later if >> needed. >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> Cheers, >> >>> > > > > > >>> > >> Micah >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou < >> >>> > > > > [email protected] >> >>> > > > > > > >> >>> > > > > > >>> > wrote: >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > Hello, >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > Apologies if the question was already asked, but >> >>> should >> >>> > we >> >>> > > > > care >> >>> > > > > > >>> about >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more >> efficient >> >>> than >> >>> > > > > FLOAT16 >> >>> > > > > > + >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > Regards >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > Antoine. >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on >> >>> the two >> >>> > > > PR's >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo) >> >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes >> >>> > > > > > >>> > >> > > 2) (parquet-format repo) >> >>> > > > > > >>> > >> > >> https://github.com/apache/parquet-format/pull/557 >> >>> > > > > > >>> > >> > > >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed >> something) >> >>> > > comments >> >>> > > > on >> >>> > > > > > >>> the two >> >>> > > > > > >>> > >> > PR's. >> >>> > > > > > >>> > >> > > >> >>> > > > > > >>> > >> > > Best >> >>> > > > > > >>> > >> > > Prateek >> >>> > > > > > >>> > >> > > >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR < >> >>> > > > > > >>> [email protected]> >> >>> > > > > > >>> > >> wrote: >> >>> > > > > > >>> > >> > > >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah. >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the >> >>> > > implementations` >> >>> > > > > > >>> > >> > >> For the c++ I have already started addressing >> the >> >>> > > > > feedback, I >> >>> > > > > > >>> > should >> >>> > > > > > >>> > >> be >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday. >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good >> progress >> >>> on >> >>> > the >> >>> > > > Java >> >>> > > > > > >>> > >> > implementation. >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> Best >> >>> > > > > > >>> > >> > >> Prateek >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb < >> >>> > > > > > >>> > [email protected] >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > >> wrote: >> >>> > > > > > >>> > >> > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I >> >>> will >> >>> > try >> >>> > > > and >> >>> > > > > > look >> >>> > > > > > >>> > into >> >>> > > > > > >>> > >> the >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this >> next >> >>> > week >> >>> > > > > > >>> > >> > >>> >> >>> > > > > > >>> > >> > >>> [1]: >> >>> https://github.com/apache/arrow-rs/pull/9372 >> >>> > > > > > >>> > >> > >>> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah >> >>> Kornfield < >> >>> > > > > > >>> > >> > [email protected]> >> >>> > > > > > >>> > >> > >>> wrote: >> >>> > > > > > >>> > >> > >>> >> >>> > > > > > >>> > >> > >>>> Hi Andrew, >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback >> on >> >>> at >> >>> > > least >> >>> > > > > the >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've >> waited >> >>> > till >> >>> > > > they >> >>> > > > > > are >> >>> > > > > > >>> > >> close to >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I >> >>> agree >> >>> > we >> >>> > > > are >> >>> > > > > > very >> >>> > > > > > >>> > >> close. >> >>> > > > > > >>> > >> > >>>> >> >>> > > > > > >>> > >> > >>>> -Micah >> >>> > > > > > >>> > >> > >>>> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb < >> >>> > > > > > >>> [email protected] >> >>> > > > > > >>> > > >> >>> > > > > > >>> > >> > wrote: >> >>> > > > > > >>> > >> > >>>> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek, >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me >> like >> >>> we >> >>> > are >> >>> > > > > ready >> >>> > > > > > >>> to >> >>> > > > > > >>> > >> start a >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should >> >>> postpone it >> >>> > > for >> >>> > > > > > >>> longer? >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to >> >>> review? >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> Andrew >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK >> GAUR < >> >>> > > > > > >>> > [email protected]> >> >>> > > > > > >>> > >> > >>>>> wrote: >> >>> > > > > > >>> > >> > >>>>> >> >>> > > > > > >>> > >> > >>>>>> Hi team, >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a >> chance >> >>> to >> >>> > work >> >>> > > > > > >>> through all >> >>> > > > > > >>> > >> the >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec >> doc. >> >>> Here >> >>> > > are >> >>> > > > > the >> >>> > > > > > >>> new >> >>> > > > > > >>> > >> > artifacts >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document : >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > >> >>> > > > > > >>> >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo : >> >>> > > > > > >>> > >> > >>>>>> >> >>> > https://github.com/apache/parquet-format/pull/557 >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo : >> >>> > > > > > >>> > >> > >>>>>> >> >>> > https://github.com/apache/arrow/pull/48345/changes >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java >> repo : >> >>> Work >> >>> > > for >> >>> > > > > > >>> Vinoo and >> >>> > > > > > >>> > >> > Julien >> >>> > > > > > >>> > >> > >>>>>> >> >>> > https://github.com/apache/parquet-java/pull/3397 >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking >> artifacts in >> >>> > > > > > >>> parquet-testing >> >>> > > > > > >>> > >> repo : >> >>> > > > > > >>> > >> > >>>>>> >> >>> > https://github.com/apache/parquet-testing/pull/100 >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> And >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in >> >>> progress >> >>> > > > > > >>> implementation >> >>> > > > > > >>> > in >> >>> > > > > > >>> > >> Go. >> >>> > > > > > >>> > >> > >>>>>> >> >>> https://github.com/apache/arrow-go/pull/704 >> >>> > (I >> >>> > > > > > haven't >> >>> > > > > > >>> > >> started >> >>> > > > > > >>> > >> > >>>>>> looking at it yet) >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned >> that >> >>> > this >> >>> > > > work >> >>> > > > > > is >> >>> > > > > > >>> also >> >>> > > > > > >>> > >> in >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!) >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation * >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by >> Antoine to >> >>> > > report >> >>> > > > > the >> >>> > > > > > >>> > numbers >> >>> > > > > > >>> > >> as >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have >> >>> given 1 >> >>> > > > round >> >>> > > > > of >> >>> > > > > > >>> > >> feedback >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note >> >>> that >> >>> > the >> >>> > > > > > default >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and >> not >> >>> > Q3. I >> >>> > > > got >> >>> > > > > > >>> around >> >>> > > > > > >>> > >> 70% >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding >> speed >> >>> > when >> >>> > > > > using >> >>> > > > > > >>> the O3 >> >>> > > > > > >>> > >> > flag. >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working >> with >> >>> > Vinoo >> >>> > > > and >> >>> > > > > > >>> Julien) >> >>> > > > > > >>> > >> and >> >>> > > > > > >>> > >> > **Cross >> >>> > > > > > >>> > >> > >>>>>> Language testing* >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions >> or >> >>> > > > feedback. >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — >> Spotify >> >>> > Columns >> >>> > > > > > >>> (Graviton >> >>> > > > > > >>> > 3, >> >>> > > > > > >>> > >> ARM >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1) >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ >> -O3 >> >>> > > (MB/s) │ >> >>> > > > > > >>> Speedup │ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ >> >>> 5,523 >> >>> > > │ >> >>> > > > > > >>> 1.75x │ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ >> >>> 5,685 >> >>> > > │ >> >>> > > > > > >>> 1.76x │ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ >> >>> 5,652 >> >>> > > │ >> >>> > > > > > >>> 1.77x │ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ >> >>> 5,473 >> >>> > > │ >> >>> > > > > > >>> 1.72x │ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘ >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK >> GAUR >> >>> < >> >>> > > > > > >>> > [email protected] >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > >>>>>> wrote: >> >>> > > > > > >>> > >> > >>>>>> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <[email protected]> >> : >> >>> Got >> >>> > > it. >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <[email protected]> >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start >> >>> moving >> >>> > the >> >>> > > > > spec >> >>> > > > > > >>> > >> development >> >>> > > > > > >>> > >> > >>>>>>>> into >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for >> >>> finalizing >> >>> > > it? >> >>> > > > > > >>> > >> > >>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of >> the >> >>> > > > examples I >> >>> > > > > > >>> have in >> >>> > > > > > >>> > >> the >> >>> > > > > > >>> > >> > >>>>>>> spec based >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we >> should >> >>> be >> >>> > good >> >>> > > > to >> >>> > > > > go >> >>> > > > > > >>> for >> >>> > > > > > >>> > the >> >>> > > > > > >>> > >> > >>>>>>> markdown format. >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone! >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew >> >>> > > > > > >>> > >> > >>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK >> >>> GAUR < >> >>> > > > > > >>> > >> [email protected]> >> >>> > > > > > >>> > >> > >>>>>>>> wrote: >> >>> > > > > > >>> > >> > >>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team, >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. >> >>> My PR >> >>> > > did >> >>> > > > > add >> >>> > > > > > >>> all >> >>> > > > > > >>> > the >> >>> > > > > > >>> > >> > test >> >>> > > > > > >>> > >> > >>>>>>>> files I >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. >> Maybe >> >>> we >> >>> > can >> >>> > > > club >> >>> > > > > > it >> >>> > > > > > >>> > >> together. >> >>> > > > > > >>> > >> > >>>>>>>> WIll also >> >>> > > > > > >>> > >> > >>>>>>>>> aid >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust >> >>> > > > implementation. >> >>> > > > > > >>> This is >> >>> > > > > > >>> > >> > great. >> >>> > > > > > >>> > >> > >>>>>>>> Thanks >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the >> >>> numbers >> >>> > on >> >>> > > > AMD. >> >>> > > > > > >>> Looks >> >>> > > > > > >>> > >> like >> >>> > > > > > >>> > >> > you >> >>> > > > > > >>> > >> > >>>>>>>> are >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding >> performance of >> >>> > BSS. >> >>> > > > This >> >>> > > > > > is >> >>> > > > > > >>> > >> > amazing!!. >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the >> >>> sampling >> >>> > > > design. >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In >> >>> some >> >>> > > crude >> >>> > > > > > >>> > >> experiments I >> >>> > > > > > >>> > >> > >>>>>>>> didn't get >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on >> >>> Graviton3 >> >>> > > (but >> >>> > > > > > maybe >> >>> > > > > > >>> > there >> >>> > > > > > >>> > >> was >> >>> > > > > > >>> > >> > >>>>>>>> something >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation). >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception >> >>> encoding for >> >>> > > the >> >>> > > > > > spec >> >>> > > > > > >>> in >> >>> > > > > > >>> > >> this >> >>> > > > > > >>> > >> > >>>>>>>> case. >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved >> for >> >>> all >> >>> > > open >> >>> > > > > > >>> questions >> >>> > > > > > >>> > >> minus >> >>> > > > > > >>> > >> > >>>>>>>> the >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on >> >>> this >> >>> > > soon) >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does >> >>> allow >> >>> > > for >> >>> > > > > > using >> >>> > > > > > >>> > >> FastLane >> >>> > > > > > >>> > >> > >>>>>>>> with the >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for >> layout. We >> >>> > > should >> >>> > > > be >> >>> > > > > > >>> able to >> >>> > > > > > >>> > >> > inject >> >>> > > > > > >>> > >> > >>>>>>>> any >> >>> > > > > > >>> > >> > >>>>>>>>> layout >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design. >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open >> >>> > comments >> >>> > > > on >> >>> > > > > > the >> >>> > > > > > >>> spec >> >>> > > > > > >>> > >> this >> >>> > > > > > >>> > >> > >>>>>>>> week. >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> Best >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve >> >>> > Loughran < >> >>> > > > > > >>> > >> > >>>>>>>> [email protected]> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote: >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah >> >>> Kornfield >> >>> > < >> >>> > > > > > >>> > >> > >>>>>>>> [email protected]> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote: >> >>> > > > > > >>> > >> > >>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue >> >>> described for >> >>> > > ORC >> >>> > > > > in >> >>> > > > > > >>> the >> >>> > > > > > >>> > >> paper >> >>> > > > > > >>> > >> > >>>>>>>> is that >> >>> > > > > > >>> > >> > >>>>>>>>> it >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a >> batch. >> >>> This >> >>> > > is >> >>> > > > > > >>> different >> >>> > > > > > >>> > >> then >> >>> > > > > > >>> > >> > >>>>>>>> the >> >>> > > > > > >>> > >> > >>>>>>>>> design >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still >> fixed >> >>> > > > encoding >> >>> > > > > > per >> >>> > > > > > >>> page >> >>> > > > > > >>> > >> in >> >>> > > > > > >>> > >> > >>>>>>>> parquet. >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't >> >>> think >> >>> > > > branch >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should >> >>> > > > > > >>> > >> > >>>>>>>>>> be a >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree >> >>> that >> >>> > we >> >>> > > > > should >> >>> > > > > > >>> be >> >>> > > > > > >>> > >> > >>>>>>>> conservative in >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings. >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1 >> >>> > > > > > >>> > >> > >>>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>>> >> >>> > > > > > >>> > >> > >>>>>>> >> >>> > > > > > >>> > >> > > >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> > >> >>> > > > > > >>> > >> >> >>> > > > > > >>> > > >> >>> > > > > > >>> > >> >>> > > > > > >>> >> >>> > > > > > >> >> >>> > > > > > >> >>> > > > > >> >>> > > > >> >>> > > >> >>> > >> >>> >> >> >> >
