divjotarora commented on code in PR #609: URL: https://github.com/apache/parquet-format/pull/609#discussion_r3890402466
########## Encodings.md: ########## @@ -77,13 +77,17 @@ endian integer, followed by the bytes. The dictionary encoding builds a dictionary of values encountered in a given column. The dictionary will be stored in a dictionary page per column chunk. The values are stored as integers using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size -or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is -written first, before the data pages of the column chunk. +or number of distinct values, the writer may stop using dictionary encoding and fall back to another +valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded +in the data page header's `encoding` field; readers must use that field to determine whether each +data page is dictionary encoded or uses another encoding. The dictionary page is written first, +before the data pages of the column chunk. Review Comment: The new wording implies that the first N pages of a column chunk can be dictionary-encoded and all other pages after need to be non-dictionary (specifically the "fall back to another valid encoding for subsequent data pages"). However, IIUC it is valid to interleave non-dict- and dict-encoded pages within a column chunk as long as only one dictionary is used. If you agree, maybe simpler wording could be something like: > Within a column chunk, the dictionary page is written first, before the data pages of the column chunk. After the dictionary page, data pages can use dictionary encodings or another valid encoding for the column's data type. Dictionary and non-dictionary encoded data pages may be interleaved. Readers must use each page header's `encoding` field to determine how the page should be decoded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
