prtkgaur opened a new pull request, #3784: URL: https://github.com/apache/parquet-java/pull/3784
<!-- Thanks for opening a pull request! If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose and format pull request title as below: GH-${GITHUB_ISSUE_ID}: ${SUMMARY} or simply use the title below if it is a minor issue: MINOR: ${SUMMARY} --> ### Rationale for this change FSST (Fast Static Symbol Table) is a proposed Parquet encoding for string/binary columns (parquet-format issue #531). It has a C++ reference implementation on an Arrow branch and a WIP draft in arrow-rs, but no Java implementation exists — no file, commit, issue, PR, or branch on the proposal author's fork mentions it as of 2026-09-08. A parquet-format change needs reference implementations in the ecosystem's major languages before it can be approved, so the empty Java slot is one of the things keeping the proposal at Draft. This PR fills that slot. Two follow-ups are planned and shape the design from the start: FSST_16 (16-bit codes, already required by the interop conformance fixtures) and OnPair. Both attach as new implementations behind the seams this PR adds, not as refactors of this PR's code. ### What changes are included in this PR? Scope is **parquet-column only**, mirroring how PFOR encoding was landed: no new Thrift page type, no footer fields, no `ParquetFileWriter`/`ParquetFileReader` plumbing. `ParquetMetadataConverter.getEncoding()` still throws for `Encoding.FSST`, so a `.parquet` file cannot be written with an FSST column that no released reader can open while the format proposal remains unratified. The property defaults to `false`. - **The codec port** (`values/symboltable/fsst/`), ported from CWI's reference implementation (`cwida/fsst` @ `89f49c580c6388acf3b6ed2a49e1bfde6c05e616`, MIT — the same commit Arrow vendored): the trainer's five-round gain loop, the split-plane `Counters`, the deterministic sampler, and the scalar 8-bit compressor. AVX-512 and the native table export/import are dropped; Parquet defines its own table serialization. - **Codec-agnostic seams** (`values/symboltable/`): `SymbolTable`, `SymbolTableTrainer`, `SampleReservoir`, `CodeStreamEncoder`/`Decoder`, `SymbolTablePayload` — shared by FSST now, and by FSST_16/OnPair when they land. - **Column wiring**: `SymbolTableValuesWriter`/`Reader` implementing `ValuesWriter`/`ValuesReader`, reusing `FallbackValuesWriter`/`RequiresFallback` unchanged so a column falls back to dictionary/PLAIN when FSST doesn't shrink the page, the `usesSymbolTable()`/`getSymbolTableBasedValuesReader(...)` hooks on `Encoding`, the enabling `ColumnProperty`, and the factory slot (dictionary's fallback writer, so low-cardinality columns keep RLE_DICTIONARY and FSST takes the high-cardinality tail). - **The symbol-table transport**: `SymbolTablePage` beside `DictionaryPage`, `default` methods added to `PageReader`/`PageWriter` (so no existing implementor breaks), with the publish moved onto the writer's chunk-finalize hook rather than the hot `getBytes()` path. This is also where a real defect got fixed along the way: a column that fell back to PLAIN/dictionary used to still publish a symbol table nobody would read. This transport is intentionally the seam the eventual Thrift `SYMBOL_TABLE_PAGE` page type attaches to — nothing above it should need to change once parquet-format ratifies the proposal. ### Are these changes tested? - **Byte-exactness against the C++ format.** Symbol table bodies and data page payloads extracted from the interop file in `apache/parquet-testing` PR #121 and committed as parquet-column test fixtures. Java matches C++ byte-for-byte on all five extracted chunks, including 255-symbol saturation, chunk-boundary crossings, and all-empty values. - **Encode-direction check** for the packed offset section: decoding each fixture's offsets and re-encoding them through the same `DeltaBinaryPackingValuesWriterForInteger` path the writer uses matches the fixture's own bytes byte-for-byte on all 26 packed-offset pages across the five cases. - **End-to-end through real column machinery**: values written through a `ColumnWriteStoreV1` and read back through `ColumnReadStoreImpl`/`ColumnReader`, covering multi-page chunks, nulls/empty strings, values that expand under FSST (must fall back, with no symbol table page published), both offset encodings, and a row-group boundary retraining the table. A negative control (strip the symbol table page from a hand-built page store) fails at `getColumnReader(path)` itself, confirming the test would catch a missing table rather than silently falling back to PLAIN. - **Ratio parity on real corpora.** Round-tripped byte-exact on ~10M values across 20 real string columns (TPC-H, ClickBench, and variant-JSON derived corpora, ~305 MB raw). Compressed/raw ratios ranged 0.14 (repeated short strings) to 0.96 (short, high-entropy address columns, where FSST buys almost nothing) — 0.36 overall. - **Benchmark** against DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and RLE_DICTIONARY, each with and without a zstd second pass. On a synthetic corpus with realistic repeated structure (URL templates, repeated vocabulary): | encoding | ratio | ratio+zstd | |---|---|---| | FSST | 0.208 | 0.119 | | DELTA_BYTE_ARRAY | 0.878 | 0.178 | | DELTA_LENGTH_BYTE_ARRAY | 1.012 | 0.178 | | RLE_DICTIONARY | 1.087 | 1.087 | ### Are there any user-facing changes? Yes, opt-in only. A new `ParquetProperties` flag (`fsstEnabled`) and matching `ParquetOutputFormat` conf key turn FSST on for BINARY columns, following the same per-column property pattern used for PFOR. It defaults to `false`, so existing writers and files are unaffected. `PageReader`/`PageWriter` gain new `default` methods, so no existing implementor of those interfaces needs to change. **Please Note** There is no file-format change: a file written with FSST enabled uses only existing Parquet page types, and `ParquetMetadataConverter` deliberately throws for `Encoding.FSST` in the Thrift-facing path, so this cannot silently produce a file that a released reader would fail to open through the normal encoding-mapping route. <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
