CurtHagenlocher opened a new pull request, #121: URL: https://github.com/apache/parquet-testing/pull/121
FSST (parquet-format#531) is in its final phases and needs working implementations to validate that implementers read the wire format the same way. These files are one implementation's output, offered for that purpose, in the same spirit as the ALP files in #100 and #119. Two files, identical data, differing only in compression: the symbol table page carries its own is_compressed flag, so a reader that ignores it passes one file and fails the other. Six columns per row group, all holding the same values: a reference pair encoded with DELTA_LENGTH_BYTE_ARRAY, an FSST pair (8-bit codes) and an FSST_16 pair (16-bit codes). A decoder bit-compares the FSST columns against the reference columns, so the files need no expected data alongside them. Four row groups, each training its own symbol table: URLs, NULLs and empty strings, escape-forcing values, and long values. Across the file that gives 104 data pages, 88 with DELTA_BINARY_PACKED offset arrays and 16 with PLAIN, and 110 escape sequences. The README records the two places the file reflects a reading of the proposal rather than its explicit text — the PageHeader field id for symbol_table_page_header, and the FSST_16 escape framing — so that a diverging implementation knows where to look. It also records the one gap: no FSST_16 column contains an escape, because the generating implementation's 16-bit tables always hold all 256 single-byte symbols. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
