etseidl commented on code in PR #12322:
URL: https://github.com/apache/arrow-rs/pull/12322#discussion_r4647711616
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,40 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data. Chunks also extend to record boundaries, so a
column
+/// with large repeated values can pass it by more than that. Lowering the
write
+/// batch size keeps pages closer to their limits.
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit)
+/// can also make pages smaller than the row count limit alone would. When a
batch
+/// would not fit the byte budget the writer splits it further, so an `INT64`
+/// column with a 10,000 byte page limit holds pages to about 1,250 rows even
with
+/// a much larger write batch size.
+///
+/// There is one exception to the batch size rule. When the definition and
+/// repetition levels of a batch are all the same value, for example a nullable
+/// column whose batch happens to contain no nulls, there is no level array to
+/// split and a larger batch costs no extra work. Those batches are written
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// rows at a time instead, so for them that limit is the enforcement
granularity
+/// and the write batch size does not move page boundaries.
Review Comment:
```suggestion
/// rows at a time instead, so for them that limit is the base chunk size,
/// subject to further byte-budget splitting, and the write batch size does
/// not move page boundaries.
```
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,40 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data. Chunks also extend to record boundaries, so a
column
+/// with large repeated values can pass it by more than that. Lowering the
write
+/// batch size keeps pages closer to their limits.
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit)
+/// can also make pages smaller than the row count limit alone would. When a
batch
+/// would not fit the byte budget the writer splits it further, so an `INT64`
+/// column with a 10,000 byte page limit holds pages to about 1,250 rows even
with
+/// a much larger write batch size.
Review Comment:
```suggestion
/// would not fit the byte budget the writer splits it further. For example,
/// a flat, non-null `INT64` column with dictionary encoding disabled and
/// `PLAIN` encoding can produce pages of about 1,250 rows at a 10,000 byte
/// page limit, even with a much larger write batch size.
```
Sorry, I was imprecise before. The default dictionary encoder would have
different behavior.
##########
parquet/src/file/properties.rs:
##########
@@ -229,6 +229,40 @@ enum OffsetIndexSetting {
/// Some(Encoding::PLAIN)
/// );
/// ```
+///
+/// # Page limits and batch size
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit),
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// and
[`set_dictionary_page_size_limit`](WriterPropertiesBuilder::set_dictionary_page_size_limit)
+/// are best effort, so a page can end up larger than the limit set for it.
Limits
+/// are only checked between batches, and
[`set_write_batch_size`](WriterPropertiesBuilder::set_write_batch_size)
+/// sets how large those batches are. A page can therefore pass its limit by
up to
+/// one batch worth of data. Chunks also extend to record boundaries, so a
column
+/// with large repeated values can pass it by more than that. Lowering the
write
+/// batch size keeps pages closer to their limits.
+///
+///
[`set_data_page_size_limit`](WriterPropertiesBuilder::set_data_page_size_limit)
+/// can also make pages smaller than the row count limit alone would. When a
batch
+/// would not fit the byte budget the writer splits it further, so an `INT64`
+/// column with a 10,000 byte page limit holds pages to about 1,250 rows even
with
+/// a much larger write batch size.
+///
+/// There is one exception to the batch size rule. When the definition and
+/// repetition levels of a batch are all the same value, for example a nullable
+/// column whose batch happens to contain no nulls, there is no level array to
+/// split and a larger batch costs no extra work. Those batches are written
+///
[`set_data_page_row_count_limit`](WriterPropertiesBuilder::set_data_page_row_count_limit)
+/// rows at a time instead, so for them that limit is the enforcement
granularity
+/// and the write batch size does not move page boundaries.
+///
+/// Which of the two applies depends on the write path and on the data being
+/// written, not only on the schema. The Arrow writer can pass levels in
compact
+/// form, while level slices supplied by the caller are always kept in full, so
+/// `write_batch_size` keeps its usual effect on that path. Even within one
path
+/// the same column can be batched at either size, depending on whether a given
+/// batch contains a null. A column with no level data at all, such as a
+/// non-nullable flat column, is always batched at the write batch size.
Review Comment:
```suggestion
/// non-nullable flat column, always uses the write batch size as its base
/// chunk size, again subject to further byte-budget splitting.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]