alamb commented on code in PR #609: URL: https://github.com/apache/parquet-format/pull/609#discussion_r4085385818
########## Encodings.md: ########## @@ -76,14 +76,19 @@ endian integer, followed by the bytes. ### Dictionary Encoding (PLAIN_DICTIONARY = 2 and RLE_DICTIONARY = 8) The dictionary encoding builds a dictionary of values encountered in a given column. The dictionary will be stored in a dictionary page per column chunk. The values are stored as integers -using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size -or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is -written first, before the data pages of the column chunk. +using the [RLE/Bit-Packing Hybrid](#RLE) encoding. Within a column chunk, the dictionary page is +written first, before the data pages of the column chunk. After the dictionary page, data pages can +use dictionary encoding or another valid encoding for the column's data type; for example, a writer +may stop using dictionary encoding if the dictionary grows too big, whether in size or number of +distinct values. Dictionary-encoded and non-dictionary-encoded data pages may be interleaved within Review Comment: Do we know if any writers actually interleave dictionary encoded pages with non dictionary pages? Do readers support this? This seems a reasonable statement to me, but I am not sure if it expanding the range of data that is valid parquet 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
