Hello everyone, I wanted to provide a quick update on the FSST encoding proposal for Parquet.
Over the last several months, we have incorporated the feedback received from the community and finalized the design. Thank you to everyone who reviewed the proposal and helped resolve the open questions. At this point, we are planning to move the proposal from the Design/Draft phase to the Implementation phase. Before doing so, I would like to ask for one final round of feedback and ensure that everyone who wanted to review the specification has had an opportunity to do so. Implementation work is also underway: - Devan Benz has started an Arrow Rust <https://github.com/apache/arrow-rs/pull/10153> implementation. - I have an existing Arrow C++ <https://github.com/apache/arrow/pull/48232> PoC for FSST. For the initial interoperability work, it would be particularly helpful to have implementations in Parquet Java and Arrow Go as well. If anyone is interested in owning or contributing to the Parquet Java or Arrow Go implementation, please let us know. Contributions to the existing Rust and C++ implementations and to cross language interoperability testing would also be greatly appreciated. Artifacts: - FSST specification: https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit - Tracking issue: https://github.com/apache/parquet-format/issues/531 - Original FSST mailing-list discussion: https://lists.apache.org/thread/15w66hvm516641btrljmtqhb4m38wn3s - FSST and OnPair discussion: https://lists.apache.org/thread/hfoltdl6o6txc3zp4680nns1mh29h0r8 - Arrow C++ PoC: https://github.com/apache/arrow/pull/48232 - Arrow Rust implementation: https://github.com/apache/arrow-rs/pull/10153 Based on any new comments/questions, the next steps will be to: 1. Address any feedback from this final review. 2. Open a PR updating the proposal status from Draft/PoC to Implementation. 3. Continue the java/go implementations and establish cross language tests. 4. Prepare for the formal vote once the implementation requirements have been met. Please share any remaining design feedback or implementation interest on this thread or the spec doc. Thanks again to everyone involved for the thoughtful reviews, benchmarks, and implementation work. I really appreciate all the help in bringing the proposal to this stage. Warm regards, Arnav
