Hi all,
I'd like to share a status update on the ALP encoding effort and get a feel
from the community before starting a formal vote.
We now have the specification plus implementations in both languages:
- Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
specification
- C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
encoding
- Java: apache/parquet-java#3397 — Parquet Java ALP
Implementation (by Vinoo)
Current state:
- The C++ implementation has been through 3-4 rounds of review.
- The Java implementation has been tracking well and has had initial
reviews from contributors, with no major remaining issues.
- Cross-language compatibility tests are passing: the Arrow C++ decoder
reads Java-written data bit-exactly across ~1.56M values and 18
fixtures, covering V1 and V2 pages, multiple vector sizes, and several
real datasets — zero mismatches.
As raised in the java-pr we will be adding coverage for extreme values
(those requiring
63-64 bits after FOR is applied) before we close a vote. We'll aim to get
that
done in parallel.
Plan: unless there are objections, I intend to start a vote at the end of
this week/early
next week.
Thanks
Prateek
On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <[email protected]> wrote:
> Hi Team,
>
> Just wanted to provide some updates on ALP.
> Micah and I have done 3-4 rounds of review for the c++ PR :
> https://github.com/apache/arrow/pull/48345
>
> For the Java implementation Vinoo has been working on following PR :
> https://github.com/apache/parquet-java/pull/3397
>
> Best
> Prateek
>
> On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <[email protected]>
> wrote:
>
>> Hi Antoine,
>>
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>>
>> It was. We thought we could defer it for the following reasons:
>> 1. It's not clear there are a lot of easy reference datasets to test its
>> effectiveness. It does look like there might be one or two on huggingface
>> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> 2. It seemed likely that float 16 was more likely used for values that
>> were less likely to reduce to decimal values.
>> 3. It could be added as an extension later if needed.
>>
>> Cheers,
>> Micah
>>
>>
>> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <[email protected]> wrote:
>>
>> >
>> > Hello,
>> >
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > > 1) (c++ arrow repo)
>> https://github.com/apache/arrow/pull/48345/changes
>> > > 2) (parquet-format repo)
>> > https://github.com/apache/parquet-format/pull/557
>> > >
>> > > I have addressed all (unless I missed something) comments on the two
>> > PR's.
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <[email protected]>
>> wrote:
>> > >
>> > >> Thanks Andrew and Micah.
>> > >>
>> > >> `fair amount of feedback on at least the implementations`
>> > >> For the c++ I have already started addressing the feedback, I should
>> be
>> > >> done with that Monday/Tuesday.
>> > >> I think Vinoo too has been making good progress on the Java
>> > implementation.
>> > >>
>> > >> Best
>> > >> Prateek
>> > >>
>> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <[email protected]
>> >
>> > >> wrote:
>> > >>
>> > >>> Got it. Thank you for the clarification -- I will try and look into
>> the
>> > >>> spec and the Rust implementation[1] in this next week
>> > >>>
>> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >>>
>> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > [email protected]>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Andrew,
>> > >>>> I think there is a fair amount of feedback on at least the
>> > >>>> implementations, typically I think we've waited till they are
>> close to
>> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> close.
>> > >>>>
>> > >>>> -Micah
>> > >>>>
>> > >>>> On Saturday, April 25, 2026, Andrew Lamb <[email protected]>
>> > wrote:
>> > >>>>
>> > >>>>> Thanks Prateek,
>> > >>>>>
>> > >>>>> I think from this content it looks to me like we are ready to
>> start a
>> > >>>>> vote to explicitly accept ALP into Parquet
>> > >>>>>
>> > >>>>> Does anyone know of a reason we should postpone it for longer?
>> > >>>>> Perhaps someone needs some more time to review?
>> > >>>>>
>> > >>>>> Andrew
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <[email protected]>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi team,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Hope everyone is doing well. I got a chance to work through all
>> the
>> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > artifacts
>> > >>>>>>
>> > >>>>>> 1) Spec document :
>> > >>>>>>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >>>>>>
>> > >>>>>> 2) Spec document in parquet format repo :
>> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >>>>>>
>> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >>>>>>
>> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> > Julien
>> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >>>>>>
>> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>> repo :
>> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> And
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - Go : Arnav just submitted an in progress implementation in
>> Go.
>> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> started
>> > >>>>>> looking at it yet)
>> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
>> in
>> > >>>>>> progress (So 4 languages!)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Arrow C++ implementation *
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> The PR is out and was also used by Antoine to report the numbers
>> as
>> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> feedback
>> > >>>>>> and I'm addressing them today. Please note that the default
>> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
>> 70%
>> > >>>>>> performance improvement in the decoding speed when using the O3
>> > flag.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>> and
>> > **Cross
>> > >>>>>> Language testing*
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Let me know if you have any questions or feedback.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Now pasting some performance numbers
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>> ARM
>> > >>>>>> Neoverse V1)
>> > >>>>>>
>> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >>>>>>
>> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> > >>>>>>
>> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >>>>>>
>> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> > >>>>>>
>> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> > >>>>>>
>> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> > >>>>>>
>> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> > >>>>>>
>> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <[email protected]
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> @Micah Kornfield <[email protected]> : Got it.
>> > >>>>>>>
>> > >>>>>>> @Andrew Lamb <[email protected]>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> Do you think it would be good to start moving the spec
>> development
>> > >>>>>>>> into
>> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
>> the
>> > >>>>>>> spec based
>> > >>>>>>> on the updated header size. Then we should be good to go for the
>> > >>>>>>> markdown format.
>> > >>>>>>>
>> > >>>>>>> Thanks everyone!
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Andrew
>> > >>>>>>>>
>> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> [email protected]>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi team,
>> > >>>>>>>>>
>> > >>>>>>>>> 1) Andrew
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
>> > test
>> > >>>>>>>> files I
>> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> together.
>> > >>>>>>>> WIll also
>> > >>>>>>>>> aid
>> > >>>>>>>>> cross language testing
>> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
>> > great.
>> > >>>>>>>> Thanks
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 2) Antoine
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
>> like
>> > you
>> > >>>>>>>> are
>> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > amazing!!.
>> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> experiments I
>> > >>>>>>>> didn't get
>> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
>> was
>> > >>>>>>>> something
>> > >>>>>>>>> wrong with my implementation).
>> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
>> this
>> > >>>>>>>> case.
>> > >>>>>>>>> - Awesome I think we have solved for all open questions
>> minus
>> > >>>>>>>> the
>> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 3) Micah
>> > >>>>>>>>>
>> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> FastLane
>> > >>>>>>>> with the
>> > >>>>>>>>> configurable enum value for layout. We should be able to
>> > inject
>> > >>>>>>>> any
>> > >>>>>>>>> layout
>> > >>>>>>>>> in the current design.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Working on resolving all remaining open comments on the spec
>> this
>> > >>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> Best
>> > >>>>>>>>> Prateek
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >>>>>>>> [email protected]>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >>>>>>>> [email protected]>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> paper
>> > >>>>>>>> is that
>> > >>>>>>>>> it
>> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
>> then
>> > >>>>>>>> the
>> > >>>>>>>>> design
>> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
>> in
>> > >>>>>>>> parquet.
>> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >>>>>>>> misprediction should
>> > >>>>>>>>>> be a
>> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >>>>>>>> conservative in
>> > >>>>>>>>>>> general for adding new encodings.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>> +1
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >
>> >
>> >
>> >
>>
>
