Thanks Arnav, I read the spec in detail again and I think the FSST symbol table placement is looking good to me. My main concern is that the spec is overly complicated to implement on decode.
I understand the primary motivation for adding FSST-like encodings is cases where existing codecs don't do well -- namely fast decoding speed and random access. There appear to be several features in the current spec that trade off decoding speed and random access to improve the compression ratio (e.g. length_histogram in the symbol table encoding, and delta encoding offsets, vs a simpler but less space efficient versions) I think we need to carefully evaluate if that is the right tradeoff (as parquet can already get very high compression using zstd and other general purpose compression algorithms) Andrew On Wed, Aug 12, 2026 at 2:45 AM Arnav Balyan <[email protected]> wrote: > Hello everyone, > > I wanted to provide a quick update on the FSST encoding proposal for > Parquet. > > Over the last several months, we have incorporated the feedback received > from the community and finalized the design. Thank you to everyone who > reviewed the proposal and helped resolve the open questions. > > At this point, we are planning to move the proposal from the Design/Draft > phase to the Implementation phase. Before doing so, I would like to ask for > one final round of feedback and ensure that everyone who wanted to review > the specification has had an opportunity to do so. > > Implementation work is also underway: > - Devan Benz has started an Arrow Rust > <https://github.com/apache/arrow-rs/pull/10153> implementation. > - I have an existing Arrow C++ <https://github.com/apache/arrow/pull/48232 > > > PoC for FSST. > > For the initial interoperability work, it would be particularly helpful to > have implementations in Parquet Java and Arrow Go as well. > > If anyone is interested in owning or contributing to the Parquet Java or > Arrow Go implementation, please let us know. Contributions to the existing > Rust and C++ implementations and to cross language interoperability testing > would also be greatly appreciated. > > Artifacts: > > - FSST specification: > > > https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit > > - Tracking issue: > https://github.com/apache/parquet-format/issues/531 > > - Original FSST mailing-list discussion: > https://lists.apache.org/thread/15w66hvm516641btrljmtqhb4m38wn3s > > - FSST and OnPair discussion: > https://lists.apache.org/thread/hfoltdl6o6txc3zp4680nns1mh29h0r8 > > - Arrow C++ PoC: > https://github.com/apache/arrow/pull/48232 > > - Arrow Rust implementation: > https://github.com/apache/arrow-rs/pull/10153 > > Based on any new comments/questions, the next steps will be to: > 1. Address any feedback from this final review. > 2. Open a PR updating the proposal status from Draft/PoC to Implementation. > 3. Continue the java/go implementations and establish cross language tests. > 4. Prepare for the formal vote once the implementation requirements have > been met. > > Please share any remaining design feedback or implementation interest on > this thread or the spec doc. > > Thanks again to everyone involved for the thoughtful reviews, benchmarks, > and implementation work. I really appreciate all the help in bringing the > proposal to this stage. > > Warm regards, > Arnav >
