Hi Team, Just wanted to provide some updates on ALP. Micah and I have done 3-4 rounds of review for the c++ PR : https://github.com/apache/arrow/pull/48345
For the Java implementation Vinoo has been working on following PR : https://github.com/apache/parquet-java/pull/3397 Best Prateek On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <[email protected]> wrote: > Hi Antoine, > > > Apologies if the question was already asked, but should we care about > > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 + > > BYTE_STREAM_SPLIT + LZ4 for example? > > It was. We thought we could defer it for the following reasons: > 1. It's not clear there are a lot of easy reference datasets to test its > effectiveness. It does look like there might be one or two on huggingface > (e.g. https://huggingface.co/datasets/kikitora/curdie). > 2. It seemed likely that float 16 was more likely used for values that > were less likely to reduce to decimal values. > 3. It could be added as an extension later if needed. > > Cheers, > Micah > > > On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <[email protected]> wrote: > > > > > Hello, > > > > Apologies if the question was already asked, but should we care about > > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 + > > BYTE_STREAM_SPLIT + LZ4 for example? > > > > Regards > > > > Antoine. > > > > > > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit : > > > Thanks Andrew and Micah for review feedback on the two PR's > > > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes > > > 2) (parquet-format repo) > > https://github.com/apache/parquet-format/pull/557 > > > > > > I have addressed all (unless I missed something) comments on the two > > PR's. > > > > > > Best > > > Prateek > > > > > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <[email protected]> > wrote: > > > > > >> Thanks Andrew and Micah. > > >> > > >> `fair amount of feedback on at least the implementations` > > >> For the c++ I have already started addressing the feedback, I should > be > > >> done with that Monday/Tuesday. > > >> I think Vinoo too has been making good progress on the Java > > implementation. > > >> > > >> Best > > >> Prateek > > >> > > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <[email protected]> > > >> wrote: > > >> > > >>> Got it. Thank you for the clarification -- I will try and look into > the > > >>> spec and the Rust implementation[1] in this next week > > >>> > > >>> [1]: https://github.com/apache/arrow-rs/pull/9372 > > >>> > > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield < > > [email protected]> > > >>> wrote: > > >>> > > >>>> Hi Andrew, > > >>>> I think there is a fair amount of feedback on at least the > > >>>> implementations, typically I think we've waited till they are close > to > > >>>> mergeable before a final vote. Otherwise I agree we are very close. > > >>>> > > >>>> -Micah > > >>>> > > >>>> On Saturday, April 25, 2026, Andrew Lamb <[email protected]> > > wrote: > > >>>> > > >>>>> Thanks Prateek, > > >>>>> > > >>>>> I think from this content it looks to me like we are ready to > start a > > >>>>> vote to explicitly accept ALP into Parquet > > >>>>> > > >>>>> Does anyone know of a reason we should postpone it for longer? > > >>>>> Perhaps someone needs some more time to review? > > >>>>> > > >>>>> Andrew > > >>>>> > > >>>>> > > >>>>> > > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <[email protected]> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi team, > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Hope everyone is doing well. I got a chance to work through all > the > > >>>>>> remaining feedback and update the spec doc. Here are the new > > artifacts > > >>>>>> > > >>>>>> 1) Spec document : > > >>>>>> > > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit > > >>>>>> > > >>>>>> 2) Spec document in parquet format repo : > > >>>>>> https://github.com/apache/parquet-format/pull/557 > > >>>>>> > > >>>>>> 3) Alp implementation in arrow c++ repo : > > >>>>>> https://github.com/apache/arrow/pull/48345/changes > > >>>>>> > > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and > > Julien > > >>>>>> https://github.com/apache/parquet-java/pull/3397 > > >>>>>> > > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing > repo : > > >>>>>> https://github.com/apache/parquet-testing/pull/100 > > >>>>>> > > >>>>>> > > >>>>>> And > > >>>>>> > > >>>>>> > > >>>>>> - Go : Arnav just submitted an in progress implementation in > Go. > > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't > started > > >>>>>> looking at it yet) > > >>>>>> - Rust : I remember Andrew mentioned that this work is also in > > >>>>>> progress (So 4 languages!) > > >>>>>> > > >>>>>> > > >>>>>> *Arrow C++ implementation * > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> The PR is out and was also used by Antoine to report the numbers > as > > >>>>>> reported here. Micah and Konstantin have given 1 round of feedback > > >>>>>> and I'm addressing them today. Please note that the default > > >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70% > > >>>>>> performance improvement in the decoding speed when using the O3 > > flag. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and > > **Cross > > >>>>>> Language testing* > > >>>>>> > > >>>>>> > > >>>>>> Let me know if you have any questions or feedback. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> Now pasting some performance numbers > > >>>>>> > > >>>>>> > > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, > ARM > > >>>>>> Neoverse V1) > > >>>>>> > > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐ > > >>>>>> > > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │ > > >>>>>> > > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤ > > >>>>>> > > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │ > > >>>>>> > > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │ > > >>>>>> > > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │ > > >>>>>> > > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │ > > >>>>>> > > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘ > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <[email protected]> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> @Micah Kornfield <[email protected]> : Got it. > > >>>>>>> > > >>>>>>> @Andrew Lamb <[email protected]> > > >>>>>>> > > >>>>>>> > > >>>>>>>> Do you think it would be good to start moving the spec > development > > >>>>>>>> into > > >>>>>>>> markdown format, in preparation for finalizing it? > > >>>>>>>> > > >>>>>>> > > >>>>>>> Yes I'll update the numbers for some of the examples I have in > the > > >>>>>>> spec based > > >>>>>>> on the updated header size. Then we should be good to go for the > > >>>>>>> markdown format. > > >>>>>>> > > >>>>>>> Thanks everyone! > > >>>>>>> > > >>>>>>> > > >>>>>>>> > > >>>>>>>> Andrew > > >>>>>>>> > > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR < > [email protected]> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi team, > > >>>>>>>>> > > >>>>>>>>> 1) Andrew > > >>>>>>>>> > > >>>>>>>>> - Thanks for working on test files. My PR did add all the > > test > > >>>>>>>> files I > > >>>>>>>>> used to benchmark on datasets. Maybe we can club it > together. > > >>>>>>>> WIll also > > >>>>>>>>> aid > > >>>>>>>>> cross language testing > > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is > > great. > > >>>>>>>> Thanks > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> 2) Antoine > > >>>>>>>>> > > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like > > you > > >>>>>>>> are > > >>>>>>>>> getting 8X the decoding performance of BSS. This is > > amazing!!. > > >>>>>>>>> - Thanks for acknowledging the sampling design. > > >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments > I > > >>>>>>>> didn't get > > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there > was > > >>>>>>>> something > > >>>>>>>>> wrong with my implementation). > > >>>>>>>>> - Locking the 16bit exception encoding for the spec in this > > >>>>>>>> case. > > >>>>>>>>> - Awesome I think we have solved for all open questions > minus > > >>>>>>>> the > > >>>>>>>>> version byte :). (will get back on this soon) > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> 3) Micah > > >>>>>>>>> > > >>>>>>>>> - FastLanes : The current spec does allow for using > FastLane > > >>>>>>>> with the > > >>>>>>>>> configurable enum value for layout. We should be able to > > inject > > >>>>>>>> any > > >>>>>>>>> layout > > >>>>>>>>> in the current design. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Working on resolving all remaining open comments on the spec > this > > >>>>>>>> week. > > >>>>>>>>> > > >>>>>>>>> Best > > >>>>>>>>> Prateek > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran < > > >>>>>>>> [email protected]> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield < > > >>>>>>>> [email protected]> > > >>>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> It looks like the actual issue described for ORC in the paper > > >>>>>>>> is that > > >>>>>>>>> it > > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different > then > > >>>>>>>> the > > >>>>>>>>> design > > >>>>>>>>>>> proposed here where there is still fixed encoding per page in > > >>>>>>>> parquet. > > >>>>>>>>>>> Given reasonably sized pages I don't think branch > > >>>>>>>> misprediction should > > >>>>>>>>>> be a > > >>>>>>>>>>> big issue for new encodings. I agree that we should be > > >>>>>>>> conservative in > > >>>>>>>>>>> general for adding new encodings. > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>> +1 > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > > > > > > > > >
