Hi team, Just wanted to resurface the doc in your email folders, looking forward to some suggestions. I'll work towards adding a few more datasets to the comparison in the coming days.
Best Prateek On Wed, Jul 1, 2026 at 9:01 AM PRATEEK GAUR <[email protected]> wrote: > Hi Antoine, > > Apologies for the delay. Got some time to work on it and updated the doc > with the requested numbers. > > - Obtained the numbers for ZSTD again. I agree they were incorrect. > - Fixed the algorithm to use decompressed size/data > - Added BSS + ZSTD to the evaluation metrics > - Added BSS + Lz4 to the evaluation metrics > > > Updated doc : > https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0#heading=h.uzgoevv9ajp > Will be pushing the branch/code for the same today. > > Best > Prateek > > On Wed, Dec 10, 2025 at 11:30 AM Antoine Pitrou <[email protected]> > wrote: > >> >> Hi, >> >> I looked at the doc and the stated decompression speeds for ZSTD look >> highly irrealistic. >> >> I think what happens is that you are computing decompression speed as: >> >> size of compressed data / time to decompress >> >> while you should really compute it as: >> >> size of uncompressed data / time to decompress >> >> Otherwise you're simply making ZSTD look miserable because it compresses >> so well. >> >> Also, I think you should also add BYTE_STREAM_SPLIT + ZSTD into the mix >> (and possible BYTE_STREAM_SPLIT + LZ4 if you're going to evaluate LZ4). >> >> Regards >> >> Antoine. >> >> >> Le 06/12/2025 à 23:24, PRATEEK GAUR a écrit : >> > Hi team, >> > >> > We wanted to share performance numbers for one of the candidate >> encodings >> > we have been discussing for parquet, for numeric compression. >> > >> > Following doc, PFOR : Encoding >> > < >> https://docs.google.com/document/d/1ZZOtxmq6K8pNU0npijfSglTJVkspXL5GLDKPSGj9HlA/edit?tab=t.0 >> >, >> > talks through numbers on compression speed, compression ratio and >> > decompression speed for columns values from the clickbench data. Based >> on >> > the numbers PFOR gives superior decompression speed compared to >> > DELTABITPACK and RLEBITPACKHYBRID and fares better on average in >> > compression ratio. Comparison has also been made with ZSTD compression >> in >> > the above doc. >> > >> > We plan to expand to a few more datasets and work on a prototype in >> arrow >> > cpp code in early Jan. >> > Looking forward to the feedback from the group. >> > >> > Best >> > Prateek >> > >> >> >>
