Okay, I sent out the voting email yesterday and so far I've heard from 3 people that it ended up in their "spam folder". Dev VPN to be blamed for it?.
What's the protocol here? I let people hunt for the email in their spam folder or send out another one? Best Prateek On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <[email protected]> wrote: > Hi team, > > Vinoo has added the tests which were requested in the last round of review. > In his own words. > > " > The extreme value tests Micah wanted are done and pushed. I added coverage > for values that need the full FOR bit width after the frame of reference is > applied: 64-bit (the case where the signed max minus min subtraction > overflows), 63-bit (the non-overflow case), and 32-bit for floats. All > lossless with zero exceptions. I also verified them through Arrow C++ ALP > reader (the local build), so the Java-written extreme columns decode > correctly in C++, bit-exact against the expected values." > > State of the PR (#3397): > > - Merges cleanly with master (I pulled master in and resolved the > conflicts). > - 41 of 44 review threads resolved. The 3 remaining are intentional: > one deferred perf nit, one that stays open until parquet-format ships ALP, > > " > > With this both the > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes > 2) Java PR : https://github.com/apache/parquet-java/pull/3397 > > Are cross language tested and range of test scenarios. > > Best > Prateek > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <[email protected]> wrote: > >> Thanks Matt, >> >> I'll be on it this week. >> >> Best >> Prateek >> >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <[email protected]> >> wrote: >> >>> There is also the Go implementation submitted by Arnav ( >>> https://github.com/apache/arrow-go/pull/704) which is waiting for >>> updates. >>> >>> Just wanted to make sure it didn't get lost here. >>> >>> --Matt >>> >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <[email protected]> >>> wrote: >>> >>> > Hi all, >>> > >>> > I'd like to share a status update on the ALP encoding effort and get a >>> feel >>> > from the community before starting a formal vote. >>> > >>> > We now have the specification plus implementations in both languages: >>> > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding >>> > specification >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add >>> ALPpd >>> > encoding >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP >>> > Implementation (by Vinoo) >>> > >>> > Current state: >>> > >>> > - The C++ implementation has been through 3-4 rounds of review. >>> > - The Java implementation has been tracking well and has had >>> initial >>> > reviews from contributors, with no major remaining issues. >>> > - Cross-language compatibility tests are passing: the Arrow C++ >>> decoder >>> > reads Java-written data bit-exactly across ~1.56M values and 18 >>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and >>> > several >>> > real datasets — zero mismatches. >>> > >>> > As raised in the java-pr we will be adding coverage for extreme values >>> > (those requiring >>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to >>> get >>> > that >>> > done in parallel. >>> > >>> > Plan: unless there are objections, I intend to start a vote at the end >>> of >>> > this week/early >>> > next week. >>> > >>> > Thanks >>> > Prateek >>> > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <[email protected]> >>> wrote: >>> > >>> > > Hi Team, >>> > > >>> > > Just wanted to provide some updates on ALP. >>> > > Micah and I have done 3-4 rounds of review for the c++ PR : >>> > > https://github.com/apache/arrow/pull/48345 >>> > > >>> > > For the Java implementation Vinoo has been working on following PR : >>> > > https://github.com/apache/parquet-java/pull/3397 >>> > > >>> > > Best >>> > > Prateek >>> > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield < >>> [email protected]> >>> > > wrote: >>> > > >>> > >> Hi Antoine, >>> > >> >>> > >> > Apologies if the question was already asked, but should we care >>> about >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 + >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >>> > >> >>> > >> It was. We thought we could defer it for the following reasons: >>> > >> 1. It's not clear there are a lot of easy reference datasets to >>> test >>> > its >>> > >> effectiveness. It does look like there might be one or two on >>> > huggingface >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie). >>> > >> 2. It seemed likely that float 16 was more likely used for values >>> that >>> > >> were less likely to reduce to decimal values. >>> > >> 3. It could be added as an extension later if needed. >>> > >> >>> > >> Cheers, >>> > >> Micah >>> > >> >>> > >> >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <[email protected]> >>> > wrote: >>> > >> >>> > >> > >>> > >> > Hello, >>> > >> > >>> > >> > Apologies if the question was already asked, but should we care >>> about >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 + >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example? >>> > >> > >>> > >> > Regards >>> > >> > >>> > >> > Antoine. >>> > >> > >>> > >> > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : >>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's >>> > >> > > 1) (c++ arrow repo) >>> > >> https://github.com/apache/arrow/pull/48345/changes >>> > >> > > 2) (parquet-format repo) >>> > >> > https://github.com/apache/parquet-format/pull/557 >>> > >> > > >>> > >> > > I have addressed all (unless I missed something) comments on >>> the two >>> > >> > PR's. >>> > >> > > >>> > >> > > Best >>> > >> > > Prateek >>> > >> > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR < >>> [email protected]> >>> > >> wrote: >>> > >> > > >>> > >> > >> Thanks Andrew and Micah. >>> > >> > >> >>> > >> > >> `fair amount of feedback on at least the implementations` >>> > >> > >> For the c++ I have already started addressing the feedback, I >>> > should >>> > >> be >>> > >> > >> done with that Monday/Tuesday. >>> > >> > >> I think Vinoo too has been making good progress on the Java >>> > >> > implementation. >>> > >> > >> >>> > >> > >> Best >>> > >> > >> Prateek >>> > >> > >> >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb < >>> > [email protected] >>> > >> > >>> > >> > >> wrote: >>> > >> > >> >>> > >> > >>> Got it. Thank you for the clarification -- I will try and look >>> > into >>> > >> the >>> > >> > >>> spec and the Rust implementation[1] in this next week >>> > >> > >>> >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372 >>> > >> > >>> >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield < >>> > >> > [email protected]> >>> > >> > >>> wrote: >>> > >> > >>> >>> > >> > >>>> Hi Andrew, >>> > >> > >>>> I think there is a fair amount of feedback on at least the >>> > >> > >>>> implementations, typically I think we've waited till they are >>> > >> close to >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very >>> > >> close. >>> > >> > >>>> >>> > >> > >>>> -Micah >>> > >> > >>>> >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb < >>> [email protected] >>> > > >>> > >> > wrote: >>> > >> > >>>> >>> > >> > >>>>> Thanks Prateek, >>> > >> > >>>>> >>> > >> > >>>>> I think from this content it looks to me like we are ready >>> to >>> > >> start a >>> > >> > >>>>> vote to explicitly accept ALP into Parquet >>> > >> > >>>>> >>> > >> > >>>>> Does anyone know of a reason we should postpone it for >>> longer? >>> > >> > >>>>> Perhaps someone needs some more time to review? >>> > >> > >>>>> >>> > >> > >>>>> Andrew >>> > >> > >>>>> >>> > >> > >>>>> >>> > >> > >>>>> >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR < >>> > [email protected]> >>> > >> > >>>>> wrote: >>> > >> > >>>>> >>> > >> > >>>>>> Hi team, >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work >>> through all >>> > >> the >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the >>> new >>> > >> > artifacts >>> > >> > >>>>>> >>> > >> > >>>>>> 1) Spec document : >>> > >> > >>>>>> >>> > >> > >>> > >> >>> > >>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit >>> > >> > >>>>>> >>> > >> > >>>>>> 2) Spec document in parquet format repo : >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557 >>> > >> > >>>>>> >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo : >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes >>> > >> > >>>>>> >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for >>> Vinoo and >>> > >> > Julien >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397 >>> > >> > >>>>>> >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in >>> parquet-testing >>> > >> repo : >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100 >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> And >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> - Go : Arnav just submitted an in progress >>> implementation >>> > in >>> > >> Go. >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't >>> > >> started >>> > >> > >>>>>> looking at it yet) >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is >>> also >>> > >> in >>> > >> > >>>>>> progress (So 4 languages!) >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> *Arrow C++ implementation * >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> The PR is out and was also used by Antoine to report the >>> > numbers >>> > >> as >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of >>> > >> feedback >>> > >> > >>>>>> and I'm addressing them today. Please note that the default >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got >>> around >>> > >> 70% >>> > >> > >>>>>> performance improvement in the decoding speed when using >>> the O3 >>> > >> > flag. >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and >>> Julien) >>> > >> and >>> > >> > **Cross >>> > >> > >>>>>> Language testing* >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> Let me know if you have any questions or feedback. >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> Now pasting some performance numbers >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns >>> (Graviton >>> > 3, >>> > >> ARM >>> > >> > >>>>>> Neoverse V1) >>> > >> > >>>>>> >>> > >> > >>>>>> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ >>> > >> > >>>>>> >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ >>> Speedup │ >>> > >> > >>>>>> >>> > >> > >>>>>> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤ >>> > >> > >>>>>> >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │ >>> 1.75x │ >>> > >> > >>>>>> >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │ >>> 1.76x │ >>> > >> > >>>>>> >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │ >>> 1.77x │ >>> > >> > >>>>>> >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │ >>> 1.72x │ >>> > >> > >>>>>> >>> > >> > >>>>>> >>> └──────────────────┴──────────────┴──────────────┴─────────┘ >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR < >>> > [email protected] >>> > >> > >>> > >> > >>>>>> wrote: >>> > >> > >>>>>> >>> > >> > >>>>>>> @Micah Kornfield <[email protected]> : Got it. >>> > >> > >>>>>>> >>> > >> > >>>>>>> @Andrew Lamb <[email protected]> >>> > >> > >>>>>>> >>> > >> > >>>>>>> >>> > >> > >>>>>>>> Do you think it would be good to start moving the spec >>> > >> development >>> > >> > >>>>>>>> into >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it? >>> > >> > >>>>>>>> >>> > >> > >>>>>>> >>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I >>> have in >>> > >> the >>> > >> > >>>>>>> spec based >>> > >> > >>>>>>> on the updated header size. Then we should be good to go >>> for >>> > the >>> > >> > >>>>>>> markdown format. >>> > >> > >>>>>>> >>> > >> > >>>>>>> Thanks everyone! >>> > >> > >>>>>>> >>> > >> > >>>>>>> >>> > >> > >>>>>>>> >>> > >> > >>>>>>>> Andrew >>> > >> > >>>>>>>> >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR < >>> > >> [email protected]> >>> > >> > >>>>>>>> wrote: >>> > >> > >>>>>>>> >>> > >> > >>>>>>>>> Hi team, >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> 1) Andrew >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add >>> all >>> > the >>> > >> > test >>> > >> > >>>>>>>> files I >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it >>> > >> together. >>> > >> > >>>>>>>> WIll also >>> > >> > >>>>>>>>> aid >>> > >> > >>>>>>>>> cross language testing >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. >>> This is >>> > >> > great. >>> > >> > >>>>>>>> Thanks >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> 2) Antoine >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. >>> Looks >>> > >> like >>> > >> > you >>> > >> > >>>>>>>> are >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is >>> > >> > amazing!!. >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design. >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude >>> > >> experiments I >>> > >> > >>>>>>>> didn't get >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe >>> > there >>> > >> was >>> > >> > >>>>>>>> something >>> > >> > >>>>>>>>> wrong with my implementation). >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec >>> in >>> > >> this >>> > >> > >>>>>>>> case. >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open >>> questions >>> > >> minus >>> > >> > >>>>>>>> the >>> > >> > >>>>>>>>> version byte :). (will get back on this soon) >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> 3) Micah >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using >>> > >> FastLane >>> > >> > >>>>>>>> with the >>> > >> > >>>>>>>>> configurable enum value for layout. We should be >>> able to >>> > >> > inject >>> > >> > >>>>>>>> any >>> > >> > >>>>>>>>> layout >>> > >> > >>>>>>>>> in the current design. >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> Working on resolving all remaining open comments on the >>> spec >>> > >> this >>> > >> > >>>>>>>> week. >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> Best >>> > >> > >>>>>>>>> Prateek >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran < >>> > >> > >>>>>>>> [email protected]> >>> > >> > >>>>>>>>> wrote: >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield < >>> > >> > >>>>>>>> [email protected]> >>> > >> > >>>>>>>>>> wrote: >>> > >> > >>>>>>>>>> >>> > >> > >>>>>>>>>>> >>> > >> > >>>>>>>>>>> >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in >>> the >>> > >> paper >>> > >> > >>>>>>>> is that >>> > >> > >>>>>>>>> it >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is >>> different >>> > >> then >>> > >> > >>>>>>>> the >>> > >> > >>>>>>>>> design >>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per >>> page >>> > >> in >>> > >> > >>>>>>>> parquet. >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch >>> > >> > >>>>>>>> misprediction should >>> > >> > >>>>>>>>>> be a >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should >>> be >>> > >> > >>>>>>>> conservative in >>> > >> > >>>>>>>>>>> general for adding new encodings. >>> > >> > >>>>>>>>>>> >>> > >> > >>>>>>>>>>> >>> > >> > >>>>>>>>>> +1 >>> > >> > >>>>>>>>>> >>> > >> > >>>>>>>>> >>> > >> > >>>>>>>> >>> > >> > >>>>>>> >>> > >> > > >>> > >> > >>> > >> > >>> > >> > >>> > >> >>> > > >>> > >>> >>
