adriangb commented on code in PR #10505:
URL: https://github.com/apache/arrow-rs/pull/10505#discussion_r3818680041
##########
parquet/src/column/writer/mod.rs:
##########
@@ -1039,6 +1051,42 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a,
E> {
}
}
+ /// Exempt a page's mandatory first value from the data page byte limit,
+ /// when that value alone already exceeds it.
+ ///
+ /// Parquet requires every data page to hold at least one value, so such a
+ /// value cannot be split out no matter how the limit is set. Counting it
+ /// against the limit makes the limit unsatisfiable, and
+ /// `should_add_data_page` then cuts a page after every single value.
+ ///
+ /// For `DELTA_BYTE_ARRAY` that costs more than the extra pages. A value is
+ /// stored as a suffix of the value before it, and a page boundary resets
+ /// what "the value before it" refers to, so one value per page means every
+ /// value is stored in full: a column of large values sharing long prefixes
+ /// writes exactly the bytes `PLAIN` would
+ /// ([#10489](https://github.com/apache/arrow-rs/issues/10489)).
+ ///
+ /// Only encodings that compress against the preceding value opt in, so
+ /// `PLAIN` and `DELTA_LENGTH_BYTE_ARRAY` keep their tighter one-value page
+ /// bound.
+ ///
+ /// Known limitation: the caller's trigger keys on a page-opening
+ /// mini-batch holding exactly one value. Nulls in a chunk make the
+ /// byte-budget chunker emit multi-level mini-batches, so on nullable
+ /// columns pages that open with a two-value mini-batch miss the
+ /// exemption and dedup is only partial; see
+ ///
`test_column_writer_delta_byte_array_nullable_shared_prefix_partial_dedup`.
Review Comment:
that’s a test so unfortunately we can’t make it an actual link. the good
news is the test and the limitation itself will be lifted by #10554 which we
can tackle after this. so they should be short lived.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
