divjotarora commented on code in PR #609: URL: https://github.com/apache/parquet-format/pull/609#discussion_r4085456940
########## Encodings.md: ########## @@ -76,14 +76,19 @@ endian integer, followed by the bytes. ### Dictionary Encoding (PLAIN_DICTIONARY = 2 and RLE_DICTIONARY = 8) The dictionary encoding builds a dictionary of values encountered in a given column. The dictionary will be stored in a dictionary page per column chunk. The values are stored as integers -using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size -or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is -written first, before the data pages of the column chunk. +using the [RLE/Bit-Packing Hybrid](#RLE) encoding. Within a column chunk, the dictionary page is +written first, before the data pages of the column chunk. After the dictionary page, data pages can +use dictionary encoding or another valid encoding for the column's data type; for example, a writer +may stop using dictionary encoding if the dictionary grows too big, whether in size or number of +distinct values. Dictionary-encoded and non-dictionary-encoded data pages may be interleaved within Review Comment: I mentioned it in my previous comment because the format already currently allows it (i.e. it's not explicitly banned). If readers don't support it, that's a bug IMO. I don't think it should block this PR specifically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
