Thanks Prateek, I also wanted to explicitly mention here on the mailing list again how similar I think FSST and OnPair are in terms of encodings -- specifically they both define symbol tables and then encode the text as sequences of those symbols. The differences are 1. the number of bits used to encode each symbol (8 vs up to 16) 2. The algorithm used to select the tables
In other words, I think we could add an encoding to Parquet that encompasses both techniques Andrew On Mon, Aug 3, 2026 at 2:12 AM PRATEEK GAUR <[email protected]> wrote: > Hi team, > > I came across a recent string compression algorithm called OnPair and spent > some time experimenting with it as a Parquet encoding, benchmarked against > FSST, DELTA_LENGTH_BYTE_ARRAY, dictionary encoding, and the usual > zstd/lz4/snappy page compression across 30 string corpora. > > Report > <https://docs.google.com/document/d/1cctFqNVh_Ul6huiXyMeFWNaGhZba0fMJ/edit > > > > Short version: it trades encode speed for ratio and decode speed — it > decodes faster than every compressed alternative measured and wins ratio on > most text-heavy columns, but its training pass makes encode substantially > slower. > > Apologies in advance for slow replies — I'm travelling. > > Best > Prateek >
