Based on my imperfect understanding of the spec, using a PLAIN dictionary page to store the symbol table as a BYTE_ARRAY would notably increase the size of the table because each entry would need a four-byte length for a symbol that's often shorter than four bytes. The symbol table avoids this by storing a length histogram that describes all symbols regardless of length.
(It also would subtly break any tooling that currently expects dictionary pages to contain complete column values instead of fragments.) Maybe it's premature, but I've published some test data in https://github.com/apache/parquet-testing/pull/121 which I believe corresponds to the current version of the spec. On Thu, Aug 13, 2026 at 1:10 AM Gunnar Morling <[email protected]> wrote: > > Hey Arnav, > > Thanks for this proposal, this looks great overall. > > One question for my understanding: have you considered, instead of adding a > new dedicated symbol page type, re-using the existing dict page mechanism? > It seems a symbol table essentially is a specific kind of dictionary, so > I'm wondering whether it could be stored as such. > > This would reduce the surface of the spec change quite a bit, only new > encoding enums (FSST, FSST_16) plus the FSST data page layout. Writers > would have to ensure that they don't store more than 255 (or 65,535, > respectively) dictionary entries, each at most 8 (or 16) bytes. > > If it has been considered before and been decided against, it might be > worth being added under a "Rejected alternatives" appendix? > > Thanks, > > --Gunnar > > > On Thu, 13 Aug 2026 at 02:56, Arnav Balyan <[email protected]> wrote: > > > Hi Curt, > > Thanks for taking a look! You’re reading it correctly. > > The existing C++ and Rust work began while the specification was still in > > progress. > > I’m waiting for the final comments before finalizing the spec. Once the > > spec has been finalized and voted upon, the next step will be to update and > > complete the implementations so they conform to it. > > > > Thanks and Regards, > > Arnav > > > > On Thu, Aug 13, 2026 at 5:19 AM Curt Hagenlocher <[email protected]> > > wrote: > > > > > It seems to me that neither the C++ PoC nor the Rust PoC are > > > consistent with what's in the current specification. Am I reading > > > something incorrectly? > > > > > > On Tue, Aug 11, 2026 at 11:45 PM Arnav Balyan <[email protected]> > > > wrote: > > > > > > > > Hello everyone, > > > > > > > > I wanted to provide a quick update on the FSST encoding proposal for > > > > Parquet. > > > > > > > > Over the last several months, we have incorporated the feedback > > received > > > > from the community and finalized the design. Thank you to everyone who > > > > reviewed the proposal and helped resolve the open questions. > > > > > > > > At this point, we are planning to move the proposal from the > > Design/Draft > > > > phase to the Implementation phase. Before doing so, I would like to ask > > > for > > > > one final round of feedback and ensure that everyone who wanted to > > review > > > > the specification has had an opportunity to do so. > > > > > > > > Implementation work is also underway: > > > > - Devan Benz has started an Arrow Rust > > > > <https://github.com/apache/arrow-rs/pull/10153> implementation. > > > > - I have an existing Arrow C++ < > > > https://github.com/apache/arrow/pull/48232> > > > > PoC for FSST. > > > > > > > > For the initial interoperability work, it would be particularly helpful > > > to > > > > have implementations in Parquet Java and Arrow Go as well. > > > > > > > > If anyone is interested in owning or contributing to the Parquet Java > > or > > > > Arrow Go implementation, please let us know. Contributions to the > > > existing > > > > Rust and C++ implementations and to cross language interoperability > > > testing > > > > would also be greatly appreciated. > > > > > > > > Artifacts: > > > > > > > > - FSST specification: > > > > > > > > > > > > > https://docs.google.com/document/d/1Xg2b8HR19QnI3nhtQUDWZJhCLwJzW6y9tU1ziiLFZrM/edit > > > > > > > > - Tracking issue: > > > > https://github.com/apache/parquet-format/issues/531 > > > > > > > > - Original FSST mailing-list discussion: > > > > https://lists.apache.org/thread/15w66hvm516641btrljmtqhb4m38wn3s > > > > > > > > - FSST and OnPair discussion: > > > > https://lists.apache.org/thread/hfoltdl6o6txc3zp4680nns1mh29h0r8 > > > > > > > > - Arrow C++ PoC: > > > > https://github.com/apache/arrow/pull/48232 > > > > > > > > - Arrow Rust implementation: > > > > https://github.com/apache/arrow-rs/pull/10153 > > > > > > > > Based on any new comments/questions, the next steps will be to: > > > > 1. Address any feedback from this final review. > > > > 2. Open a PR updating the proposal status from Draft/PoC to > > > Implementation. > > > > 3. Continue the java/go implementations and establish cross language > > > tests. > > > > 4. Prepare for the formal vote once the implementation requirements > > have > > > > been met. > > > > > > > > Please share any remaining design feedback or implementation interest > > on > > > > this thread or the spec doc. > > > > > > > > Thanks again to everyone involved for the thoughtful reviews, > > benchmarks, > > > > and implementation work. I really appreciate all the help in bringing > > the > > > > proposal to this stage. > > > > > > > > Warm regards, > > > > Arnav > > > > >
