etseidl commented on code in PR #10775:
URL: https://github.com/apache/arrow-rs/pull/10775#discussion_r3834304695
##########
parquet/src/file/properties.rs:
##########
@@ -1443,6 +1498,73 @@ impl Default for EnabledStatistics {
}
}
+/// Controls when dictionary encoding falls back to the column's fallback
encoding.
+///
+/// While dictionary encoding is enabled for a column, the writer buffers a
+/// dictionary of the distinct values seen so far. This policy decides when the
+/// writer gives up on dictionary encoding for the remainder of the column
chunk
+/// and switches to the fallback encoding (e.g. `PLAIN`, or a delta encoding
+/// with [`WriterVersion::PARQUET_2_0`]).
+///
+/// This enum is `#[non_exhaustive]`: additional policies may be added in the
+/// future.
+#[derive(Debug, Clone, Copy, PartialEq)]
+#[non_exhaustive]
+pub enum DictionaryFallback {
+ /// Fall back as soon as the dictionary page exceeds
+ /// [`WriterProperties::dictionary_page_size_limit`].
+ ///
+ /// This is the default, and the historical behavior of this crate.
+ OnPageSizeLimit,
+ /// Keep the dictionary past
[`WriterProperties::dictionary_page_size_limit`]
Review Comment:
This dives too deep in the weeds and has become a TL;DR 😅 Please ask claude
to summarize a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
